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INTRODUCTION 



The fact that the Multi-Media Economic Analysis Course, 
which represents a radical departure from the conventional way 
economics is taught, continues to be offered at the Naval 
Academy several semesters after its introduction attests to its 
acceptability, practicality, and success. The purpose of this 
report is to present the findings obtained in evaluating several 
important aspects of the Multi-Media Economic Analysis Course 
as it was presented at the United States Naval Academy during 
the evaluation semester (Fall 1969-70). 

The following aspects of the course were evaluated: 

1. Student performance 

2. Student and instructor acceptance 

3. Course management 

4. General operational environment. 

The questions related to these topics and the findings are 
summarized below and are discussed fully later on in the report. 
It is assumed that the reader is familiar with the contents of 
TR- 5.37 (Economics Course Evaluation) and TR-5.39 (Final 
Validation Report) ; however, for convenience of the reader 
certain parts of these reports are included in Appendix A 
and Appendix B, respectively. 
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SUMMARY 



' The Multi-Media Economic Analysis Course in its fully 
operational mode was presented at the United States Naval 
Academy during the Fall Semester, 1969-70, to 39 randomly 
selected third class (sophomore) and second class (junior) 
Midshipmen who had not previously had any economics courses. 

In the course, the students proceeded at their own pace and 
exercised options in the selection of media and enrichment, 
and thus maintained a significant degree of control over when 
and how they studied. Whenever students have such freedom of 
choice,. many questions arise concerning how well they learn, 
whether they can complete the course in the required time, 
whether they like having the responsibility for their learning, 
etc. During the evaluation of the course these and similar 
questions were investigated. Both the questions investigated 
and the findings are presented below. 

1 . Did the Midshipmen achieve at the expected level of 
performance? 

Ninety-seven percent of the students achieved 80 
percent or more of the core objectives of the course. Further- 
more, 100 percent of the Midshipmen, in order to meet their 
learning contracts, accomplished varying numbers of enrichment 
objectives in addition to those in core. The Academy awarded 
a grade of "C" to students who achieved the core objectives. 
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Ninety percent of the students earned sufficient option points 
from their enrichment activities to obtain a final grade of 
"A" or "B'\ 

2. What was the relationship between a student's rank in 
test performance and his rank in *QPR, SATV and SATQ? 

The correlations between rank in test performance and 
rank in QPR, SATV and SATQ were low and positive? however, for 
five tests the correlation .between rank in QPR and rank in 
test performance was statistically significant. In the case 
of four tests the correlation between rank in SATQ and rank 
in test performance was also statistically significant. Con- 
sequently, it may be assumed that Midshipmen who rank high in 
SATQ and in QPF. also tend to rank high in tests. 
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3. Did the students show a gain in learning? 

Prom both the practical and statistical points of 
view all students showed a significant gain in learning. The 
amount of new information acquired and the increase in their 
ability to work with economic principles and concepts, as 
reflected in the difference between the scores on pretests and 
post tests, indicated a great increase in learning. In response 
to questionnaires, the Midshipmen reported that as a result of 
taking the course, they could perform better in activities such 
as discussing economic problems with parents, read with greater 
understanding newspaper stories related to economics, and 
criticize economic policy with a reasoned, stronger basis for 
their arguments. 

4. Were the Midshipmen able to complete the course in 
one semester? 

Although it is generally recognized that the Multi- 
, Media Economic Analysis Course is the equivalent of a two- 
semester course, thirty-two percent of the students completed 
the course six weeks before the end of the semester. All of 
these students earned a final grade of A or B, which required 
that they also achieve enrichment objectives. One objective 
in designing the course was to produce course segments that 
the median student in the conventional course could complete 
in about 50 minutes' learning time. The median learning time 
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per segment for the Midshipmen in all concept areas was less 
than 50 minutes, as were the mean and modal learning times. 

The mean total learning time used by the Midshipmen for all 
the core materials in the Multi-Media Economics Course was 51 
hours and 29 minutes, only 10 to 15 hours more than the time 
required for class attendance in the conventional course. 

The Naval Academy student with the greatest total learning 
time used 97 hours and 39 minutes for core materials, while 
the one with the least used 30 hours. and 33 minutes. 

Of considerable interest is the fact that both were 
among the group who completed the course in only nine calendar 
weeks. All others completed the economics course in one 
semester . 

All correlations between a Midshipman's total learning 
time per test and his score on the corresponding test, as well 
as between his rank in total learning time per post test and 
his rank in SATV, SATQ and QPR, were low. Therefore, it may be 
concluded that little or no relationship existed between total 
learning time per test and any of these variables. 

5. Did the five Midshipmen who made the highest scores 
on each control test differ significantly from those five who 
made the lowest scores in respect to mean total learning time? 
What was the correlation between rank in test scores and rank 
in learning time for each group? Was there any significant 
difference between the groups in respect to mean QPR and mean 
scores on SATV and SATQ? 
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The individuals included in the group making the five 
highest scores or in the group making the five lowest scores 
varied for every test. The mean total learning time of the 
five Midshipmen making the lowest scores was greater than that 
of the five making the highest scores in six of the nine tests; 
in the other three* the students with the highest scores had 
the greatest mean total learning time. Although the mean 
■differences between the groups were statistically significant 

i 

in only two tests* almost all were significant from a practical 
point of view because they usually amounted to an hour or more. 

A student and instructor* knowing of this difference* might be 
able to plan more efficient learning contracts. 

The correlations between rank in test scores and rank 
in total learning time for the five Midshipmen with the highest 
test scores were either zero or less than .30 in all but two 
tests. On those two* the correlations* though high and neg- 
ative (indicating a high* inverse relationship between learning 
time and test score) were not statistically significant. 
Correlations between the same variables for the groups making 
the lowest scores tended to be low and in three cases* the 
correlations were negative. These findings indicated little 
relationship between rank in test scores and rank in learning 
time among the low-scoring groups. 

In general* the Midshipmen who were among the five 
making the highest scores on each test also scored higher on 
SATV and SATQ and had higher QPR’s than the five making the 
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lowest scores. The difference in the mean QPR's of the groups 
was in all but one case: statistically significant in favor of 
those with the high scores. However, in only three or four 
tests were the mean differences in SATV and SATQ scores statis- 
tically significant in favor of the groups of high scorers. 

Since the differences amounted to approximately 50 points, 
they had practical significance also. 

6. Did the five Midshipmen who used the greatest total 
learning time differ significantly from the five who used the 
least total learning time in respect to test scores? What 
was the correlation between rank in learning time and rank in 
test scores for each group? Was there any significant 
difference between the groups in respect to mean QPR and mean 
SATV and SATQ scores? 

The five Midshipmen using the least total learning 
time usually had a mean test score slightly higher (always less 
than 2 points) than those using the greatest total learning time. 
In no test was this difference either statistically or prac- 
tically significant. These findings are typical of courses 
requiring the mastery of a high percentage of core material. 

Correlations between rank in total learning time per 
test and rank in test scores for the group using the least total 
learning time were, with one exception, low and were inclined to 
be negative. The correlation between these variables on test 
16 was -.84, indicating a high degree of inverse relationship. 
Students who took the least total learning time for this test 
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Correlations between rank 



tended to have the lowest scores, 
in total learning time .per test and rank in test scores for 
the group using the greatest total learning time also were 
generally negative and low. However, on three tests the 
correlations were +.89, +.87, and +.84, showing a high, direct 
relationship between the two rankings. Among all these 
correlations only .89 and .87 were statistically signi- 
ficant. In the case of test 11 the correlation of -.80 
indicates that students who required the greatest total learning 
time for this test tended to make the highest scores on it. 

The personnel of the group of five Midshipmen using 
the least learning time and of the group using the greatest 
learning time varied from test to test. Although the mean QPR 
and the mean SATV and SATQ scores of the groups with the least 
learning time were higher than those of the other groups, the 
difference in the means for all three variables were generally 
neither statistically nor practically significant. However, the 
mean differences in SATQ of about 50 points favoring the group 
requiring the least total learning time for tests 72, 80 and 95 
had both practical and statistical significance. 

7. Did the group of Midshipmen who completed the course 
six weeks before the end of the semester differ significantly 
from the class as a whole in total learning time, test scores, 
QPR,, SATV or SATQ scores? 

The group of Midshipmen who completed the Multi-Media 
Economics Course in nine weeks were nqt .different from the 
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class as a whole in total learning time, test scores, QPR, 

SATV or SATQ scores . They probably were different in motiva- 
tion. Both the Midshipman who used the greatest totai learning 
time for core content and the one who used the least were in 
the group. Moreover, three others who were in the groups taking 
the least total learning time per test and four who had been in 
the groups taking the greatest total learning time completed 
the course six weeks early. Eight of this group had been among 
the five with the highest test scores; four had been among the 
five with the lowest test scores. All statistical evidence 
pointed to the fact that these students were like the rest of 
the class in respect to total learning time, test scores, QPR, 
SATV scores and SATQ scores. 

8. What conclusions can be drawn from these findings 
concerning the performance of the Midshipmen in the Multir-Media 
Economic Analysis Course? 

The findings of these studies present conclvisive 
evidence that the core materials in this economics course can 
be achieved in one semester or less by Naval Academy students 
similar to those in the experimental program at a 90/80 level 
of performance and that many can complete additional enrichment 
material at an equally high performance level within this same 
time period. The self-pacing feature of the course permits 
students who require a greater total learning time than their 
classmates to complete the course in one semester or less, also. 
These results imply that the design of the instructional 



materials, which incorporated the sequencing of objectives, the 
conditions suitable for ithe type of learning inherent in each 
objective, and the reinforcement of correct responses, is sound. 
Eliminating any part of the design in revising this course could 
result in less spectacular student performance. On the basis 
of the findings concerning the performance of the Midshipmen, 
the Multi-Media Economics Course must be evaluated as highly 
successful in promoting efficient learning of economics by 
students at the Naval Academy. 

9. Did Midshipmen who used audio packages learn the 
objectives for these segments as well as they learned those 
presented in the printed tents? How did the performance of 
these Midshipmen compare with that of the rest of the class in 
respect to test scores and total learning time? 

The Midshipmen who elected to use the audio tape- 
workbook packages in general did not show great variation in 
their performance on test items referenced to the segments in 
the audio packages and pn test items referenced to segments 
immediately preceding and immediately following the audio, 
which were presented in another medium printed text. 

Comparisons of the percentage of correct items related 
to those segments prepared in alternate media for the group who 
used the audio and for the rest of the class were not signifi- 
cantly different, statistically or practically, on the whole, 
the mean total learning time of the audio group was somewhat 
less (5 - 10 minutes) than that of the rest of the class. For 



one audio package the mean total learning time of the audio 
group was 17 minutes less; for a different package it was 20 
minutes greater. These differences would be important to a 
busy Midshipman in deciding whether or not to try an audio 
package . 

10. Did the Midshipmen who viewed the films have sig- 
nificantly better scores on test items related to the films? 

Midshipmen who saw the films had a mean score on 
items referenced to them in test 27 which was one point higher 
than that of those not viewing the films. This difference was 
both practically and statistically significant. The difference 
in mean scores of viewers and non-viewers on items referenced 
to films in tests 48 and 72 was statistically but not practi- 
cally significant. 

11. How effective were the computer simulations? 

’The course included eleven computer simulations of 
economic systems. Some of these simulations are meant to 
challenge the most advanced student. No conclusions can be 
drawn concerning the effectiveness of the simulations , since 
the computer was usually down when a student needed it. With 
down time rates reportedly running as high as 80 percent for 
some students, students typically had to replan their schedules 
several times in order to run one simulation. 




11 - 



12. What was the student reaction to the course as a 
whole? What was the instructor's reaction? 

Student reaction to the course was overwhelmingly 
favorable. They found it neither too easy, nor too hard and 
preferred the self “instructional approach to the traditional 
lecture approach. When asked to tell what they liked best 
about the course, they responded most frequently that they liked 
to be able to determine when, where, and how to study and to 
decide what final grade to work for. In response to a request 
for suggestions for improvement, they asked that more non- 
' mandatory seminars relating economic principles to current 
policies and problems be held. 

The instructor, too, was highly complimentary of the 
course because it promoted the learning of .economics easily and 
well. Although he likes the lecture method, he appreciated the 
opportunity to work on a more individual basis with students 
which the self-instructional course afforded him. He, too, 
suggested there should be seminars concerning current problems 
and policies. (As the course is designed, the instructor 
determines when seminars are to be held and their topics. This 
may suggest that "required" seminars should be built into the 
course . } 

13. What was student reaction to the films? 

Reaction to the films was mixed. On the one hand, 

a majority of the students said that the films were an effective 
supplement to the course, and that they were relevant and 
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interesting? yet, a majority also reported under the category 
of what was least liked; about the films that they were boring 
and a waste of time. There were other opinions expressed 
which were just as contradictory. In view of this reaction, 
the whole question of films should be reviewed. 

14. What was student reaction to the audio tape-workbook 
packages? 

Forty percent of the Midshipmen elected to use any 
audio-package series. Two of them used all three of the series; 
the others used one or two. About eighty percent of all the 
students utilising this medium were favorably impressed because 
the tapes made learning easier for them and provided a change 
in pace. Those who disliked the tapes said they could not 
concentrate on economics while listening and found it difficult 
to go back over material they did not understand. A majority of 
those who used this medium suggested that more audio packages 
should be included in the course. 

15. What problems, if any, existed in the general oper- 
ational environment of the course? 

A self-paced, self-instructional course requires 
that all materials and media be available to any student when 
he is ready for them. Operating such a course in the normal 
environment of the traditional school posed a number of prob- 
lems in logistics. The Naval Academy solved many of these; how- 
ever, a few such as coping with computer down-time apparently 
remain unsolved. Since the films were rented, they could be 



shown only on a group, not individual, basis at a specified 
time. Consequently, the content of the films was not always 
congruent with that which the Midshipmen were learning. The 
facility in which the films were viewed and the hours when 
they could be seen were not completely satisfactory. The 
security of the space in which all materials, including control 
test and student records, were stored could not be ensured. 

Some breakdown in communication between instructor and students 
in respect to the scheduling of seminars and appointments was 
experienced. All these were problems typically connected with 
inaugurating a different type of course in any school. 

3. STUDENT POPULATION AND LIMITATIONS TO DATA 

In any detailed discussion of the Multi-Media Economic 
Analysis Course, the characteristics of the sample population 
who took the course should be considered at the outset. 
Thirty-nine Midshipmen at the United States Naval Academy, 13 
second classmen (juniors) and 26 third classmen (sophomores), 
were randomly selected by the Academy to take the course during 
the Fall semester 1969-70. None of the students reported 
having previously had a course in economics. Table 1 presents 
the scores of the whole group on the SAT Verbal and the SAT 
Quantitative, as well as their Cumulative Quality Point Rank 
(QPR) as of, 9/1/69. These data reveal a fairly wide range in 
all three measures as would be found in a random sample. The 
difference of 82 points between the mean score on the SATV and 
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the mean score on the SATQ is statistically significant at 
P < .01 in favor of the. SATQ and is probably characteristic of 
students in an institution oriented to Naval science and 
engineering such as the Naval Academy. 



TABLE 1 

The Mean, SD, Median, and Range of the QPR and of the 
Scores of 39 Midshipmen on the SATV and SATQ, 9/1/69 



Index 


. Mean 


SD 


Median 


Range 




QPR 


2.5 


.5 


2.315 


1.66 - 


3.83 


SATV 


587.4 


59.3 


580.00 


486.00 - 


719.00 


SATQ 


669.5 


54.2 


670.00 


550.00 - 


771.00 



Figures 1, 2, 3 and 4 represent frequency distributions 
of the scores on the SATV and SATQ and the QPR for the Mid- 
shipmen. Although the mean QPR for the group is 2.5 (C+) , 51 
percent of the students have QPR's between 2.0 and 2.49, which 
is average or C level performance (see Figure 3. } . 

The finding noted above, that there is a significant 
statistical difference between the mean SATV and the mean SATQ 
scores, is corroborated by the fact that, when frequency 
polygons are drawn and the means indicated on them as in 
Figure 4, the mean scores for the two tests do not fall in the 
same plane, giving rise to the assumption that the verbal and 
quantitative abilities of these students are different. 




Pig. 1 — Distribution of Score SATV 




Pig. 2 — Distribution of Scores SATQ 
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Fig. 3 — Distribution of QPR 





Fig. 4 — Distributions and Means of 
SATV and SATQ 

SATV (X) 

SATQ (Y) 
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Questions concerning the relationship of a student s 
rank in one variable with his rank in each of the other var- 
iables were also investigated. A test of the significance 
of the correlations reveals that the relationship between rank 
in SATV and rank in QPR and between rank in SATQ and QPR are 
both significant (Table 2), indicating that students who rank 
high in SATV and SATQ also tend to rank high in QPR. 



TABLE 2 

Correlations of Ranks of 39 Midshipmen on. 
SATV, SATQ and in QPR, 9/1/69 







SATV 


JIatq. 


QPR 


SATV 




— 


.28 


.38* 


SATQ 




.28 


— 


.54** 


QPR 




.38* 


.54** 


— 




*P 

**p 


< .05 

< .01 







Although the key information in this report concerns 
the performance of Midshipmen who took the course during the 
Pall Semester, 1969-70, it is worthwhile, in the interest of 
determining how representative of the Naval Academy population 
these students are, to compare their mean scores on the SATV, 
SATQ and rank in QPR at the beginning of the course with the 
mean scores and rank on the same variables of the Midshipmen 
who took the course during the validation semester. Spring 

s The rank order used was from high to low, the highest 
score receiving a rank of one. 

-18- 
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1968-69. ^ That the two samples are from the same population 
is evidenced both from inspection of the mean scores and mean 
Qpr of the two which are almost identical, as well as from a 
t-test of the significance of the difference between the means 
of the two groups in all three variables (Table 3) . 

TABLE 3 



Comparison of the Mean Scores on SATV and SATQ 
and of the Mean QPR of Two Groups of Midshipmen 
Who Took the Multi-Media Economics Course 



Index 


Mean-Group I 
Spring, 1968-9 


Me an -Group II 
Fall. 1969-70 


ft 


t-test 


SATV 


589 


587 


68 


.1413 


SATQ 


674 


670 


68 


.3145 


Qpr 


2.7 


2.5 


68 


1.4717 



The Midshipmen who took the course during the Fall 
semester, 1969-70 received a completely revised version of the 
one used during the validation phase, 1968-69. All changes 
were based on empirical evidence obtained during the validation 
trials. Although the course content remained essentially 
unchanged, some segments were rewritten; objectives were rearranged 
in segments; objectives were combined or omitted because they 
had already been presented in an earlier segment, and all tests 



It would have been interesting to compare the performance 
of the samples in these variables with the performance of the 
total population at the Naval Academy; however, the latter infor- 
mation was not available to us. 



were revised. Therefore, any comparisons of performance of 
the two groups on tests which might carry the same identifica- 
tion number would be spurious because, since the revision, the same 
objectives are not necessarily included in the two tests. Con- 
sequently, this study is generally limited to a report of what 
the Midshipmen who took the Fall, 1969 revision of the Multi- 
Media Economic s Course did . 

The collection of data was greatly facilitated during 
the evaluation semester by the course manager, who gathered 
the hundreds of Dy media cards and saw that the needed informa- 
tion was recorded on them, as a result, there were only very 
minor data gaps, the most important of which concerns the amount 
of time used for review. The students recorded the total time 
spent in reviewing a group of segments, rather than the amount 
of time spent in reviewing each segment in the group. Since 
the review time was added to the segment just preceding the 
test, the total learning time for that segment was increased 
disproportionately? however, all other data concerning learning 
time was not affected. It has been noted also that one or two 
students may have recorded as their self-test time the total 
amount of time required for both the learning of the segment and 
the taking of the self-test., rather than only the latter. This 
tended to increase the total learning time . 

2 

The number of core segments and core objectives in the 
Fall, 1969 Multi-Media Economics Course is given in Appendix C. 
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Not every student handed in a questionnaire after each 
test, and the instructor's critiques were obtained mainly from 
informal conversations rather than from the instructor's 
critique forms. Therefore , data concerning student acceptance 
is based on about 85 percent return, of questionnaires. 

Although there are the few flaws mentioned above , we 
found the data available quite complete and useful in obtaining 
meaningful analyses of the Multi-Media Economics Course , Fall 
semester 1969-70. 

4. STUDENT PERFORMANCE 

4.1 Number of Core Objectives Learned 

A 90/80 (90 percent of the students learn 80 percent 
of the core objectives) goal wa*> established for the evaluation 
semester of the course. However, not only 90 percent but 97 
percent of the Midshipmen learned 80 percent of the core objec- 
tives of the course. It should be noted that along with the 
595 explicit core objectives, of necessity, there are many 
implicit core objectives to be learned in the course* The 
latter are not represented in tests but should be recognized 
as additional learning which took place. Moreover, 100 percent 
of the Midshipmen elected as part of their learning contracts 
to take some enrichment which also entailed learning more 
objectives . How many enrichment objectives each student learned 
depended upon how much he contracted to learn. A student could 
earn option points to increase his grade by learning enrichment, 
and 90 percent of the Midshipmen earned sufficient option points 



from their enrichment experiences to be awarded a final grade 
in the course of A or In the light of the fact that 51 

percent of the Midshipmen had, prior to talcing this course, 
a QPR of C (2.0 through 2.49), having 90% of the students 
earn an A or B would seem to imply that this type of course 
motivated these students to learn. 

How many objectives a student learned was indicated 
by his score on a post test given when the student had 
completed the work of each of the four concept areas. The 
post test consists of criterion referenced questions for each 
terminal objective in the concept area. Since the course is 

hierarchical in nature, it has been assumed that correctly answer 
ing the criterion referenced question for one terminal objective 
represents the learning not only of the terminal objective itself 
but also of all the enabling objectives supporting it. Thus, if 
the Midshipmen answered correctly 95 percent of the items 
on a ; post test, it was assumed that he had learned at least 
95 percent of the terminal objectives and all the objectives 
subsumed under them. Tables 4, 5, 6 summarize the percentages 
of students in the validation semester and in the evaluation 
semester who learned 90 precent or more, 85 percent or more, 
and 80 percent or more of the core objectives. 



A grade of "C M was awarded by the Academy for successful 
completion of the core objectives of the course. A student 
could elect to earn a C, B, or A, depending on the number of 
objectives he achieved. 



TABLE 4 



The Percentage tof Midshipmen Who Learned 90S or More 
of the Core Objectives Spring Semester 1968-69 and 
Fall Semester 1969-70, by Concept Area 



Concept 

Area 


Spring Semester 
1968-1969 


Fall Semester 
1969-1970 


I 


13 


95 


II 


58 


64 


III 


17 


33 


IV 


10 


59 



For both groups of students Concept Area III seems the 
most difficult since in both the percentage of students able 
to pass 90 percent or more of the terminal objectives is small. 
This may be accounted for in part by the fact that this concept 
area has more objectives to be learned than any other concept 
area and that the suggested number of weeks for learning the 
objectives is comparatively shorter than that for any other 
concept area. The increase in the percentage passing Concept 
Area I during 1969-70 probably reflects everyone's familiarity 
with the self-instructional course and improvements resulting 
from revisions. 
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TABLE 5 



The Percentage of Midshipmen Who Learned 85% or More 
of the Core Objectives Spring Semester 1968-1969 and 
Fall Semester 1969-1970, by Concept Area 



Concept 

Area 


Spring Semester 
1968-1969 


Fall Semester 
1969-1970 


■ I 


39 


98 


II 


90 


95 


III 


57 


82 


IV 


23 


84 



The remarkable increase in the percentage of students 
who passed 85 percent or more of the objectives during the 
evaluation semester undoubtedly is due to the revisions made 
after the validation tryouts. 



TABLE 6 

The Percentage of Midshipmen Who Learned 80% or More 
of the Core Objectives Spring Semester 1968-1969 and 
Fall Semester, 1969-1970, by Concept Area 

Concept Spring Semester Fall Semester 
Area 1968-1969 1969-1970 



I 


80 


100 


II 


90 


100 


III. 


70 


100 


IV 


50 


97 



Only one student did not learn 80 percent of the core 
objectives. He learned 76 percent and reported that he could 



have learned more had he not spent so. much time working on 



enrichment. Although his pattern of falling behind the proposed 
deadlines was apparent when he finished Concept Area II a month 
beyond the suggested deadline, he did not heed the warnings 
given to him at that time. The incident suggests that in the 
future those in charge of the course should be on the lookout 
for this type of student and that learning contracts for such 

, students stipulate that core work must be satisfactorily com - 

j 

j pleted before any enrichment may be attempted. The latter course 
of action was voluntarily chosen by several of the successful 
students during the evaluation semester. 

I 

4 . 2 Test Results 

Scores on all unit and post tests follow a pattern 
usually found in courses where students are required to master 
core learnings . The mean score on each test is very close to 
the highest possible score, and the measures of dispersion 
(range, standard deviation, standard error) are generally 
quite small (see Table 7,) . 
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TABLE 7 



A Summary of Test Results for a Group of 39 
Midshipmen Who Took the Multi-Media Economics 
Course. Fall Semester 1969-70** 



Test 


Highest 








Standard Standard 




Number 


Possible 


Mode 


Median 


Mean 


Devia 


Error 


Range 




Score 








tion 






2R* 


30 


14 


13 


12 


5.1 


.82 


20(00-20} 


11T* 


39 


35 


36 


36 


1.3 


.21 


5(33-38) 


16P* 


30 


28 


28 


28 


1.8 


.28 


6(24-30) 


18R 


51 


0 


10 


11 


8.4 


1.34 


35 (00-35) 


27T 


34 


31, 33 


31 


32 


1.7 


.28 


5 (29-34) 


39T 


44 


40,42 


41 


41 


1.9 


.30 


7 (37-44) 


48P 


51 


46 


46 


46 


2.3 


.37 


9 (41-50) 


50R ' 


51 


10 


9 


10 


6.4 


1.02 


24 (00-24) 


61T 


32 


31 


31 


30 


1.3 


.21 


5 (27-32) 


72T 


38 


34 


34 


34 


1.3 


.21 


4 (32-36) 


8 OP 


51 


44 


44 


45 


2.5 


.40 


9 (41-50) 


83R 


37 


0 


3 


5 


5.7 


.91 


22 (00-22) 


95P 


37 


32,35 


33 


33 


2.1 


.34 


9(27-36} 


*R indicates a pretest; 


T — a unit test; 


P — a ] 


post test. 




**A11 


test results 


are reported 


in raw 


scores 


throughout 


the report 



Figures 5, 6, 10 and 11 are typical of the distributions 
of scores on mastery tests. Figures 7, 8, 12 and 13 illustrate 
the bi-modality of the test scores with one mode below and one 
mode above the mean in each test. The lone score of 27 shown 
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Fig. 5— Distribution of Scores Unit Test 11 



Fig. 6 — Distribution of Scores Post Test 16 





27 




Fig. 7 — Distribution of Scores Unit Test 27 




Fig. 8 — Distribution of Scores Unit Test 39 
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Fig. 9 — Distribution of Scores Post Test 48 





Fig. 10 — Distribution of Scores Unit Test 61 
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Fig. 13 — Distribution of Scores Post Test 95 



in Figure 13 belongs to the student who learned only 76 percent 
of the core objectives. All other low scores in any of Figures 
5-13 represent learning a minimum of 80 or 85 percent of the 
core objectives. 

Although there exists some correlation between rank in 
QPR and rank on SATV and SATQ, whether there exists a correlation 
between rank in these variables and rank in scores on tests 
was also investigated. Inasmuch as the self-instructional 
packages basically involve reading and comprehension skills, 
it might be assumed that- students ranking high on the SAT 
Verbal would also rank high in their test scores. It might 
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also be expected that students ranking 2 high in QPR would also 

_ +-L 

rank high in test performance. However, one of the main 
objectives of the project was to design an economics course 
such that any Midshipman, no matter what his previous ranks 
or classifications had been, could learn the core materials. 

In such a situation, there would be little relationship between 
the student's rank in SATV, SATQ, and QPR and his rank in test 
results . 

The Multi-Media Economics Course meets the above objective 
to some extent. However, there exists as shown in the case of 5 
tests, a significant correlation between rank in QPR and rank in 
test performance, and in the case of 4 tests a significant correla- 
tion between rank in SATQ and rank in test performance (see Table 8) 
Thus it would seem that students who ranked high in QPR and SATQ 
would tend also to rank high on tests. 



^ank order is from high to low with the highest 
score receiving a rank of one. 



TABLE 8 



Correlation of Rank 5n Test Performance with 
Rank in SATV, SATQ and QPR for, 

39 USNA Midshipmen, Fall 1969-70 



Test 


SATV 


SATQ 


QPR 


11T ° 


.13 


* 1 1 


.23 


16P ° 


.26 


. .38 * 


.31 


27T 


.03 


.27 


.40 * 


39T 


.20 


.55 ** 


.34 * 


4 8P 


.11 


.42** 


.54** 


61T 


.29 


.28 


.44** 


72T 


.08 


.05 


.09 


80P 


.31 


.40 * 


.36* 


95P 


.48 ** 


.13 


.22 


°T = Unit 


test; P = 


Post test 





*P < .05 
**P < . 01 



The correlations in Table 8 are very similar to those 
obtained during the validation phase (see Appendix B, p. 15). 
Obtaining similar results in both the evaluation and validation 
semesters tends to corroborate the conclusion that Midshipmen 
who rank high in SATQ and QPR also tend to rank high in tests. 
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4. 3 Gain in Learning 

Although the findings thus far indicate that the 
students learned the core objectives of the ecoromics course, 
does this learning represent a significant increase in know - 
ledge of economic concepts over that which they already pos- 
sessed prior to taking the Multi-Media Economics Course? At 
the beginning of a concept area each student took a pretest 
designed to measure what he knew about the topics included 
in that concept area. Midshipmen were cautioned on the pre- 
test to respond only to those questions for which they knew 
the answer and to leave the others blank. A study of pretest 
data shows that there was not one instance where a student 
achieved criterion performance (80 per cent) on any pretest. 

On the most elementary material contained in the first con- 
cept area, which involved the enabling objectives for all sub- 
sequent advanced objectives, no student achieved criterion 
performance. Since in no case did a student demonstrate on a 
pretest that he had mastered the objectives of a concept area, 
it was considered valid for the students to study the objectives 
Later when a Midshipman had completed learning all the seg- 
ments in the concept area, he was given a post test on those 
topics, a scrambled form of the pretest. It has been assumed 
that the difference between the scores on the two tests repre- 
sents gain in learning. 

The post test scores of the Midshipmen, both as 
individuals and as a group, are remarkably higher than the 
corresponding pretest scores, and inspection of the mean dif- 
ference (Table 9) shows a gain in learning substantial enough 



to be practical. Moreover, in their responses to questionnaires 
some students wrote that they had learned enough economics to 
enable them now to read^ newspaper articles concerning economic 
policies with more understanding than prior to taking the 
course; some that now they could discuss various topics in eco- 
nomics with their fathers; and others that their attitudes toward 
economics had become more favorable. (The questionnaires are 
discussed in greater detail later in the section "Student Accept- 
ance.") A t-test of the significance of the difference between 
the mean score of each pretest and the mean score of each post 
test is statistically significant at P<.01 (Table 9). 

Therefore, we may conclude from this and the findings 
given in the preceding paragraphs that the gain in the learning 
of economics exhibited by the Midshipmen is both practically 
and statistically significant. And we may further conclude 
from the t-test that this increase is due not to chance but 

t 

i 

t 

to some other intervening variable, in this case, to the self- 
instructional course materials. 




TABLE 9 



Comparison of the Mean Scores for Midshipmen 
on the Pretest and Post Test for Each Concept 





Area, 


Fall Semester 


1969-70 






Concept 

Area 


Pretest 

Mean 


Post Test 
Mean 


Mean 

Difference 


d f 


t-test 


I 


11.9 


2 7.9 


16.0 


76 


15.0* 


II 


10.5 


46.3 


35.8 


76 


29.5* 


III 


9.9 


. 44.6 


34.7 


76 


29.1* 


IV 


5.1 


32.7 


27.6 


76 


25.5* 



*P< .01 
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4 . 4 Learning Time Required 

"Learning ’time," as used in this report, refers to 
the number of minutes a student used in learning a segment of the 
course. Because all of the following activities, according 
to the design of the Multi-Media Economics Course, are integral 
parts of the conditions of learning, the learning time for a 
segment includes the number of minutes spent in reading the 
material related to each objective and completing the practice 
exercises embedded in it plus the time spent in taking the 
self-test and restudying objectives which, as revealed by the 
results of that test, had not been learned. The amount of time 
required for such tasks is quite different from that which would 
be spent in merely reading, scanning or covering the materials 
written about an objective. A student's total learning time 
for a test is the sum of the learning times for all segments 
included in the particular test. For example, the total learning 
time for Unit Test 11 is the sum of the student's learning time 
for segments 4, 5, 6, 7, 8, 9, and 10, while the total learning 
time for Post Test 16 represents the sum of the student's 
learning time for segments 4, 5, 6, 7, 8, 9, 10, 12, 13, 14 
and 15. The amount of time spent in taking a unit test, a 
pretest or a post test is never included in learning time. 

In section 4.1 we reported on the attainment of one of 
the objectives of the project; namely, to design a course in 
which 80 percent of the core materials could be learned by 90 
percent of the students. Another objective was to design a 
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course whose core materials could be learned in one semester 
at the above mentioned level of performance by any student, 
regardless of his learning habits and abilities. This goal 
has also been achieved. 

Thirty-two percent of the Midshipmen had completed the 
Multi-Media Economics Course by December 9, 1969, approximately 
six weeks before the end of the semester. All of these students 
had fulfilled contracts for an A or B grade, which meant that, 
in addition to learning core material, they had successfully 
completed the learning of several enrichment segments. One 
student who earned an A had completed the course as of November 
14, 1969. Since the Multi-Media Economics Course has been 
generally recognized as the equivalent of a two-semester course, 
the fact that a third of the students completed it six weeks 
before the end of the semester indicates that the design of 
this self-instructional course is highly motivating and promotes 
efficient learning; i.e., the learning of large amounts of 
materials in less than the usually required amount of time. 

Each segment has been designed to require approximately 
50 minutes of learning time for the "C" student in the con- 
ventional system; however, the Midshipmen as a group have found 
this a generous estimate. Not only is their mean learning time 
per segment less than 50 minutes but so too are the median and 
mode (number of minutes most frequently recorded by the Midship- 
men), and this occurs even though the range in learning time 
per segment is wide. Table 10 presents these data. 
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TABLE 10 



The Mean, Median, Mode and Range of the Learning 
Time (in Minutes) Used by a Group of 39 Midshipmen, 



per 


Segment 


in Each Concept 


Area 




Concept 

Area 


Mean 


Median 


Mode 


Range 


I 


49 


45. 


45 


15-180 


II 


39 


33 


25 


11-208 


III 


42 


35 


30 


10-440 


IV 


42 


38 


40 


8-150 



The broad range in learning time per segment is due 
in part to the fact that in several instances the students 
added their review time to the segment immediately preceding 
the test and also in part to the fact that some students found 
certain concepts very difficult and needed additional review 
time . ^ 

The correlation of learning time with other variables 
has been investigated. What is the degree of relationship 
between a student's rank in total learning time s for each post 
test and his rank in SATV, SATQ and QPR? It might be postulated 
that students having the highest SATV scores would require less 
learning time than other students, (thus ranking high in learning 
time), giving a high positive correlation or that students having 

The limitations to the data have been discussed in 
section 3 of this report. 

s Rank order of total learning time was from least to 
greatest, with the student who used the least total learning 
time receiving a rank of one. 
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the highest QPR would take more learning time than other 
students, giving a high; negative correlation between these 
variables. However, the correlations between rank in total 
learning time per post test and rank in SATV, SATQ arid QPR 
are low and not significant except for test 48 (Table 11). It may be 
concluded, therefore, that in general little or no relationship 
exists between a Midshipman's rank in the total learning time for 
each post test and his rank in either SATV, SATQ or QPR. Hence, 
how much learning time a certain type of student will need is not 
predictable from these data. 

TABLE 11 

Correlation of Rank in SATV, SATQ and QPR with 
Rank in Learning Time Required for Post Tests 



for 


39 Midshipmen, 


Fall 1969 




Post Test 


SATV 


SATQ 


QPR 


16 


.24 


.13 


.13 


48 


.1*6 


.35* 


.12 


80 


.20 


.28 


.04 


95 


r* * 08 


.25 


.04 



* P <.05 



Does this same degree of relationship hold true between 
test scores and total amount of learning time used per test? 

Can it be said with any degree of certainty that students who 
spend the greatest amount of time in learning the core objectives 
score high on tests? The correlations between a Midshipman's 
total learning time per test and his corresponding test score 
are also very low and tend to be negative (Table 12). Therefore, 



it may be concluded that there is little or no relationship 
between these variables. Also, this slight relationship tends 
to be inverse, indicating that those Midshipmen who take the 
least amount of learning time tend to score high on tests and 
vice versa. However, the correlations are so low that the only 
conclusion which can be reached with any degree of certainty is 
that there is little or no relationship between these variables. 
Consequently, no predications concerning the relationship 
between learning time and test performance can be made from 
these data. 

TABLE 12 

Correlations of Test Scores with Total Learning 
Time for that Test 



Test 



11T* 

16P* 

27T 
39T 
48P 
6 IT 
72T 
80P 
95P 

*T = Unit test; p = post test 



r 

-.16 

-.13 

.02 

-.19 

-.01 

-.05 

-.27 

.01 

-.05 



Since students, instructors and others when planning 
their work are usually; interested in approximately how much 
learning time will be needed to complete learning the segments 
for a test, the mean total learning time required by the group 
of Midshipmen per control test has been determined. Inasmuch 
as post tests include items from each segment in an entire 
concept area, the mean total learning time for these tests 
is considerably larger than that for unit tests. 

Although there are more objectives in all of Concept 
Area III, tested by 80P, than in all of Concept Area II, tested 
by 48P, students took on the average about an hour less total 
learning time for Concept Area III than for Concept Area II. 

On the other hand, however, in Concept Area III there was a 
smaller percentage of students who learned 90 percent and 
85 percent of the core objectives than in Concept Area II 
(see Tables 4 and 5) . All students passed 48P on the first 
try, whereas three students did not pass 80P on the first 
try. It is pertinent to note that during a 1 fall semester 
Concept Area III is usually studied in the period between 
Thanksgiving and Christmas, and in a spring semester in the 
weeks around Easter. Both are times when motivation to study 
economics may be difficult for students to maintain, and this 
may account in part for the above findings. 
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TABLE 13 



The Mean, SD, Standard Error, and Range of Total Learning Time 
(in Minutes) Used by 39 Midshipmen, Fail 1969, per Control Test 



Test 

Number 


No. of Segments 
Tested 


No. of Objec- 
tives Tested 


Mean Total 
Learning Time 


SD 


SE 


Range 


11T* 


7 


81 


374 


87.9 


14 


400 (255-655) 








;6 , i4 n ) 


(1'28") 




( 6 1 4 0 " ) 


16P* 


11 


122 


535 


132.1 


21.2 


661 (349-1010) 








(8 ' 55 ") 


(2*12") 




(11*1") 


27T 


9 


60 


298 


94.0 


15.1 


370 (148-518) 








(4* 58") 


(1 * 3 4 M ) 




(6*10") 


39T 


11 


85 


456 


150.9 


24.2 


579 (241-820) 








(7 ' 36 ") 


(2*31") 




(9*39”) 


48P 


.27 


192 


1046 


326.0 


52.2 


1251 (619-1870) 








{17 1 26" ) 


( 5 1 26 " ) 




{20*51") 


61T 


9 


63 


303 


110.3 


17.7 


420 (165-585) 








{5 '3”) 


(1*50") 




<7M 


72T 


9 


74 


428 


155.7 


24.9 


433 (230-663) 








( 7 T 8 w ) 


(2* 36") 




{7*13") 


80P 


25 


204 


1003 


354.7 


56.8 


1700 (545-2245) 








(16 '43") 


(5 *55") 




(28 '20") 


95P 


12 


77 


508 


194.3 


31.1 


921 (240-1161) 








(8*28") 


(3 ’14") 




(15*21") 
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*T = Unit test? P = Post test 



The final question to be answered concerning total 



learning time is: What is the mean total learning time used 

by Midshipmen to complete all core segments in the entire 
Multi-Media Economics Course? The mean total learning time 
for this group of Midshipmen is 51 hours and 29 minutes (Table 
14). The student taking the least amount of total learning 
time spent 30 hours and 33 minutes, while the student taking 
the greatest amount used 97 hours and 39 minutes, a difference 
of 67 hours and 6 minutes. It should be noted again that 
each Midshipman upon completing a segment recorded the number 
of minutes he used in learning it, and the above results were 
obtained from the students' figures. The total learning 
time does not include time spent in taking pretests, post 
tests and unit tests, which would be an additional 7 hours 
and 45 minutes. 

TABLE 14 

The Mean, SD, SE and Range of the Total Learning 
Time Used by a Group of 39 Midshipmen in Learning 
Core Objectives of the Multi-Media Economics 
Course, Fall 1969 

Mean SD SE Range 

51 '39" IS' 52" 2 ' 32 " 30'33" - 97 ' 39 " (67 ' 06 " ) 

Usually the student in a conventional economics course 
spends 36 to 45 hours per semester just attending class, and 
he is expected to spend an additional 72 to 90 hours in outside 
reading and study, a total of 108 to 135 hours for a three- 
credit course. However, the majority of students do not use 
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that much time in learning the conventional economics course, 
and the question of how much real time is used by them was not 
investigated for this report. But whether Midshipmen taking 
the greatest total learning time can in one semester finish 
the Multimedia Economics Course, including taking tests and 
completing enough enrichment activities for a final grade of A, 
has been investigated. The findings indicate that students 
can do this. Xn fact, the Midshipman who required the greatest 
total learning time and the one who required the least were 
amoving those who completed the economics course six weeks before 
the end of the semester, and both earned a final grade of A. 

When we remember that the Multi-Media Economics Course 
has been recognized as the equivalent of a two semester course, 
the fact that the Midshipmen required a mean total learning 
time for core materials of 51% hours (59% hours including testing) , 
only a few hours more than class time in a conventional lecture 
course, and that they exceeded a 90/80 performance level, is 
strong evidence of the success of this economics course in 
promoting effective and efficient learning. The self-instruc- 
tional, self-paced aspects of this course permitted the Mid- 
shipmen to meet their individual requirements as to when and 
how they would study and for what final grade they would work, 
and obviously this resulted in learning a large amount of 
material, in a comparatively short time, at a high level of 
performance. It should not be assumed, however, that since 
the total learning time for the Naval Academy students for the 




- 44 - 



core materials, including taking the tests, is on the average 
the equivalent of approximately eight 8-hour days, that we 
recommend learning the core materials in eight days or less. 

We do not. ■ The economics course abounds in fairly complex 
and abstract principles which need to be assimilated and seen 
in perspective, and participating over time in seminars or 
in enrichment activities permits assimilation and seeing 
relationships. Therefore, we recommend that students continue 
to set their own goals and their own paces, but that they be 
encouraged to consider recommended time frames when making 
their decisions in these matters. 

4 . 5 Studies of Subgroups 

To examine in depth the relationships between total 
learning time and test performance, three subgroups of Mid- 
shipmen were studied. One subgroup comprises 10 Midshipmen, 
the five who scored lowest and the five who scored highest 
on each control test; a second, also 10 Midshipmen, the five 
who took the greatest amount and the five who took the least 
amount of total learning time; and a third, of 12 Midshipmen 
who completed the Multi-Media Economics Course six weeks before 
the end of the semester. The findings related to each subgroup 
are treated separately. Since small differences in the perfor- 
mance of these subgroups may be of practical significance to 
instructors, students and administrators, we have used a .2 
probability level for the statistical test of the significance 
of the differences in means. 



4.5.1 Midshipmen with Highest Test Scores and 
with Lowest Test Scores 



The relationship between learning time and 
either high or low test performance is of major importance in 
this project. Originally, the evaluation design proposed a study 
of the relationship between the scores and learning time of 
those students scoring in the top ten per cent and bottom ten 
per cent on each test. However, at that time estimates of the 
number of students who might be chosen to take the Multi-Media 
Economics Course varied from 15 to 44, a range of 1.5 to 4.4 
students in the top or bottom ten per cent. In order to pre- 
clude the possibility of having only one or two students in a 
group and in order to keep within a small percentage range, it 
was decided to study the five Midshipmen who scored low on 
each test and the five who scored high and the five who took 
the greatest total learning time and the five who took the 
least. The first question studied was; Do the Midshipmen 
who score highest (Group H) differ significantly in mean total 
learning time per control test from the 5 who score lowest 
(Group D? The findings are summarized in Table 15. 



TABLE 15 



Comparison of the Mean Total Learning Time (Minutes) 
Used by the 5 Midshipmen Who scored Highest (Group H) 
and the 5 Who Scored Lowest (Group L) on Each Control 

Test, Fall 1969 



Test 


Total Learning Time 


Total Learning Time 


Mean 


t“test 


Number 


Group 

Mean 


H 

SD 


Group L 
Mean SD 


Difference 




11T° 


338 

(5 1 38" ) 


66 

(1*06") 


393 

(6 1 33" ) 


42 

(42") 


-55 

(55") 


1.5544* 


16P° 


562 

(9 '22") 


178 

(2 ' 58") 


674 

(11' 14") 


203 

(3 ' 23" ) 


-112 
( 1 ' 5 2 " ) 


.9306 


27T 


218 

(3 ’38") 


43 

(43") 


304 

(5 ' 04 ") 


137 

(2*17") 


-86 

(1*26") 


1.3401 


39T 


422 

(7 '02") 


63 

(1 ' 03" ) 


517 

(8*37") 


164 

(2*44") 


-95 

(1 ' 35" ) 


1.2159 


48P 


1161 
(19' 21") 


417 

(6*57") 


998 

{ 16 ' 38 ") 


286 

(4 ' 46 " ) 


+ 163 
(2*43") 


.7218 


61T 


312 

(5' 12") 


157 

(2 ' 37" ) 


289 

(4 *49") 


105 

{ 1 ' 4 5 " ) 


+23 

(23") 


.2715 


72T 


456 

( 7 ' 2 6 " ) 


114 

(1 : 54" ) 


668 

(11' 08") 


265 

(4*25") 


-212 

(3*32") 


1.7254* 


8 OP 


988 

(16*28") 


414 

(6*54") 


906 

(15*06") 


282 

(16*28") 


+ 82 

(1 ' 22 " ) 


.3641 


95P 


445 

(7*25") 


99 

(1*39") 


439 

(7*19") 


102 

( 1 ' 4 2 " ) 


-06 

(06") 


.0977 


n 

E-* 

0 


Unit test) 


P = Post 


test 









* P< .2 
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In all but three tests the lowest scoring students 
have a greater mean total learning time than the highest 
scoring students. In the three others the reverse is true. 

For unit test 61 and post test 95, the mean difference between 
the two groups is so low as to have neither practical nor 
statistical significance. However, the additional time spent 
by low scoring students in learning materials for tests 11, 16, 
27, 72 and 80 has practical significance for instructors or 
t. curse managers who, armed with such information, may be able 
to assist Midshipmen planning contracts to more realistically 
determine how much they may be able to accomplish with suggested 
time limits. In the case of tests 11 and 72 the mean difference 
is also statistically significant. 

The same students do not consistently appear in either 
group on every test. Only one Midshipman scored among the 
lowest in 7 out of 9 tests, one in 6 out of the nine, and only 
three or four in 2 or 3 of the nine. The remainder appear only 

once in this group. This is also true for those in the high- 

* 

scoring groups, with the exception that only 1 Midshipman scored 
high in 6 out of 9, and none in 7 out of 9 tests. Whether the 
group with high scores differs significantly in QPR and scores 
on SATV and SATQ from the group with low scores is a question 
raised for this investigation. In general, the Midshipmen who 
were among those making the five highest scores have higher 
QPR's and score higher in SATV and SATQ (Table 16). 
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TABLE 16 



Comparison of the Mean QPR of the 5 Midshipmen 
Who Scored Highest (Group H) and the 5 Who Scored 
Lowest (Group L) on Each Control Test, Fall 1969 



Test 


QPR- Group H 


QPR - Group L 




Mean 


t-test 


Number 


Mean 


SD 


Mean 


SD 


Difference 




11T° 


2.47 


. 37 


2.15 


.23 




.32 


1.6370* 


16P° 


2.89 


.50 


2.22 


.22 




.67 


2.7102* 


27T 


2.83 


.67 


2.01 


.25 




.82 


2.6005* 


39T 


2.91 


.54 


2.38 


.47 




.53 


1.6400* 


48P 


2.62 


.65 


1.94 


.17 




.68 


2.2875* 


61T 


2.53 


.32 


2.14 


.46 




.39 


1.5720* 


72T 


2.43 


.34 


2.22 


.38 




.21 


0.9112 


80P 


3.18 


.57 


2.13 


.10 


1 


.05 


4.0304** 


95P 


2.53 


.41 


2.02 


.08 




.51 


2.7216* 


°T = Unit test; 


P = 


Post test 










* P< . 


,2 














**P< 


. 2 and P ^ 


.01 












The difference in the mean 


QPR of 


the 


two groups is 


generally about 


.50 


(half a rank point) , 


and the 


standard 


deviations are 


large 


enough so 


that there 


is ove 


rlapping 


between groups. 


However, the 


Midshipmen 


with high scores 


tend to have a 


mean 


QPR above 


2.50 (C+ or 


better), whereas 



those with the low tend to have one nearer 2.00 (C) . In 

i 

all but one test the mean difference between groups is 
statistically signif ianct in favor of Group H . These 
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findings concerning QPR are consistent with those for the 
class as a whole, reported on Table 1 of this report- 

TABLE 17 



Comparison of the Mean SATV Score of the 5 Mid- 
shipmen Who Scored Highest (Group H) and the 5 
Who Scored Lowest (Group L) on Each Control 







Test, 


Fall 


1969 






Test 


SATV- 


•Group H 


SATV- 


■Group L 


Mean 


t-test 


Number 


Mean 


SD 


Mean 


SD 


Difference 




11T° 


549 


34 


571 


32 


-22 


1.0735 


16P° 


576 


37 


541 


62 


35 


1.0858 


27T 


583 


47 


581 


42 


02 


0.0641 


39T 


624 


83 


578 


81 


46 


0.8953 


48P 


597 


80 


574 


53 


23 


0.5165 


61T 


629 


24 


572 


52 


57 


2.2034* 


72T 


553 


35 


531 


33 


22 


1.0027 


80P 


662 


. 48 


588 


69 


74 


1.9705* 


95P 


604 


63 


543 


42 


61 


1.7987* 



°T = Unit test? P = Post test 
*P< .2 
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TABLE 18 



Comparison of the Mean SATQ Score of the 5 
Midshipmen who 'Scored Highest (Group- H) and 
the 5 Who Scored Lowest (Group L) on Each 





Control Test , Pall 


1969 






.Test 


SATQ-Group H 


SATQ-Group L 


Mean 


t-test 


Number 


Mean 


SD 


Mean 


SD 


Difference 




11T° 


646 


84 


647 


39 


-01 


-0.0338 


16P° 


702 


40 


645 


39 


57 


2.2544* 


27T 


711 


32 


648 


42 


63 


2.6886* 


39T 


728 


57 


650 


43 


78 


2.4481* 


48P 


686 


75 


634 


37 


52 


1.4061 


61T 


674 


67 


647 


49 


27 


0.7188 


72T 


670 


53 


643 


42 


27 


0.8850 


80P 


727 


20- ' 


634 


60 


93 


3.2777* 


95P 


683 


78 


644 


52 


39 


0.9253 


°T = Unit test; 
*P < .2 


P = 


Post test 








\ 

In test 11 the low- scoring 


group has 


a higher mean score 


on both the 


SATV and the 


SATQ than 


the high 


scoring group ; 


however, in 


neither 


case 


is the di 


£f erence 


of practical 


or 



statistical significance. In those tests where the mean dif- 
ference between the groups in either the SATV or the SATQ scores 
is 50 points or more, the difference is statistically significant. 
However, the mean SATV scores for both the high scorers and the 
low scorers are somewhat lower than would be expected when 
compared to the mean of the class as a whole (Table 1). On the 
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other hand, except in the case of test 11, the mean SATQ 



scores for the high and low scoring groups are consistent with 
those, reported for the class as a whole in Table 1. 

Having determined the significance of the mean difference 
for the groups in respect to total learning time, QPR, SATV 
and SATQ scares, there remains the question of how great is the 
correlation between their rank in test performance and their 
rank in learning time. 2 In both tests 39T and 80P the r's 

TABLE 19 

Correlation of Rank in Test Scores and Rank in 

Learning Time per Test for the 5 Midshipmen Who 



Scored Highest 
Lowest 


(Group H) and 
(Group L), F a n 


the 5 Who Scored 
1969 


Test Number 


Group H 


Group L 


11T° 


.00 


.40 


16P° 


.00 


.37 


27T 


.00 


.00 


3 9T 


-.82 


-.64 ■ 


48P 


.28 


-.10 


61T 


.00 


.65 


72T 


.00 


.00 


80P 


-.71 


-.43 


95P 


.07 


.42 


' °T = Unit 


test? P = Post 


test 



s The rank order for test performance was from high score 
to low score? for total learning time from least to 
greatest. The highest score was ranked 1 and the least 
total learning time also was ranked 1. 



of -.82 and -.71 respectively indicate a high degree of inverse 
relationship between test score rank and rank in learning time 
for the Midshipmen with highest scores (Table 19), although 
for this small sample neither of these correlations is sig- 
nificant at P<.05. This means that the Naval Academy students 
who scored high on these tests generally took the greatest 
amount of learning time. The correlation of -.64 in test 39 
for the group with the lowest scores is also noteworthy. 

It indicates that those Midshipmen who made the lowest scores 
on test 39 tended to use the least amount of learning time? 
whereas, the correlation of +.65 in test 61 shows a tendency 
for the same Midshipmen to use a great amount of learning 
time for that test. The very low or zero correlations obtained 

for these variables on all other tests and for both groups 
may be interpreted as indicating little or no relationship 
between test scores and learning time. 

4.5.2 Midshipmen Using the Least Learning Time and 
the Greatest Learning Time 

The question, do the test scores of the 5 
Midshipmen who used the least total learning time (Group LLT) 
per control test differ significantly from the scores of the 5 
who took the greatest total learning time (Group GLT) , was 
also studied. Table 20 summarizes the results of this investi- 
gation. 
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TABLE 20 



Comparison of the Mean Test Scores of the 5 
Midshipmen who Used the Least Learning Time 
(Group LLT) and the 5 Who Used the Greatest 
Learning Time (Group GLT) per Control Test 

Fall, 1969 



Test. 

Number 


Group 

Mean 


LLT 

SD 


Group 

Mean 


GLT 

SD 


Mean 

Difference 


t-test 


11T° 


36.4 


1.6 


34.6 


2.7 


1.8 


1.2990 


16P° 


28.2 


1.1 


27.4 


2.2 


0.8 


0.7303 


27T 


31.6 


1.9 


32.0 


1.9 


-0.4 


0.3310 


39T 


40.2 


1.1 


39.8 


2.0 


0.4 


0.3849 


48P 


45.4 


2.7 


46.8 


2.2 


-1.4 


0.9037 


61T 


30.6 


1.1 


30.6 


1.3 


0.0 


0.0000 


72T 


34.0 


0.7 


33.2 


.1.8 


0.8 


0.9300 


80P 


44.2 


2.9 


44.8 


3.1 


-0.6 


0.3128 


95P 


32.8 


2.4 


32.6 


1.1 


0.2 


0.1690 



°T = unit test? P = Post test 

In no case is the difference between the mean scores of 

I 

the groups two points; in fact, it is generally merely a fraction 
of one point. Midshipmen using the least learning time have a 
mean score slightly higher than those with the greatest learning 
time, except on test 48 for which the reverse is true. From 
both a practical and a statistical point of view, there is no 
difference in the performance of the groups, a finding which 
is to be expected in a course requiring the mastery of a high 
percentage of core material. 



One Midshipman consistently appeared in every group 
using the greatest total learning time? another appeared in 
the group for 8 out of 9 tests? and 2 students in 6 out of nine. 
Among those using the least total learning time, one Midshipman 
appeared in the group 7 out of 9 times , and 2 in 6 out of 9. 

On the whole, a very few students consistently were among those 
using the greatest or the least total learning time for a test. 
The personnel of the groups varied. In view of this fact, do 
the two groups differ significantly in respect to QPR and scores 
on SATV and SATQ? 

TABLE 21 

Comparison of the Mean QPR of the 5 Midshipmen 
Who Used the Least Total Learning Time (Group LLT) 
and' the 5 Who Used the Greatest Total Learning 
Time (Group glt) per Control Test, Fall 1969 



■ Test 
Number 


Group 

Mean 


LLT 

SD 


Group 

Mean 


GLT 

SD 


Mean 

Difference 


t-test 


11T° 


2.35 


.48 


2.23 


.40 


.12 


0.4141 


16P° 


2.35 


.48 


2.34 


.37 


.01 


0.0220 


27T 


2.68 


.83 


2.60 


.39 


.08 


0.1851 


39T 


2.53 


.82 


2.38 


.33 


.15 


0.3947 


4 8P 


2.57 


.81 


2.38 


.32 


.19 


0.4960 


61T 


2.40 


.83 


2.22 


.21 


.18 


0.4902 


72T 


2.61 


.69 


2.20 


.24 


.41 


1.2587 


8 OP 


2.61 


.69 


2.20 


.23 


.41 ' 


1.2742 


\ 

95P 


2.48 


.33 


2.20 


.24 


.28 


1.5598* 


°T = Unit 


test? P 


= Post 


test 









* P < .2 
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The mean difference in QPR between the groups is less 



than .50 (one half a rank point) in all cases (Table 21), 
and except for tests 72 and 80 there is no practical difference 
in the mean QPR of the groups. The Naval Academy students 
requiring the least total learning time for. tests 72 and 80 
have a mean which is .41 points higher than that of the other 
group, indicating that these students may tend to have a C+ or 
better average while the others tend to have a somewhat lower 
one of C. This difference, however, is not statistically 
significant. A smaller mean difference for test 95 is statis- 
tically significant, although the latter, from a practical 
point of view is not as significant as the mean differences 
occurring in tests 72 and 80; 

TABLE 22 . 

■Comparison of the Mean SATV Scores of the 5 
Midshipmen who Used the Least Total Learning 
Time (Group LLT) and the 5 who Used the 
Greatest Total Learning Time (Group GLT) per 
. Control Test , Fall 1969 



Test 

Number 


Group 

Mean 


LLT 
SD ■ 


Group 

Mean 


GLT 

SD 


Mean 

Difference 


t-test 


11T° 


594 


55 


586 


47 


08 


.0.2273 


16P° 


594 


55 


582 


53 


12 


0.3486 


27T 


634 


65 


610 


83 


24 


0.5189 


39T 


622 


52 


579 


63 


43 


1.1640 


48P 


607 


55 


579 


63 


28 


0.7469' 


6 IT 


616 


64 


553 


64 


63 


1.5765* 


72T 


590 


56 


554 


60 


36 


. 0.9876 


8 OP 


590 


56 


566 


57 


24 


0.6644 


95P 


578 


58 


566 


57 ' 


12 


0.3189 



°T = Unit test* P = Post test 

*P < .2 



/ 



Referring to Table 22, we see that for only one test 
is the difference in the mean SATV score for Group LLT signi- 
ficantly different, both practically and statistically, from 
that of Group GLT. In all other tests the mean scores are 
very similar to the mean SATV for the class as a whole (Table 1) . 

TABLE 23 

Comparison of the Mean SATQ Scores of the 5 
Midshipmen Who Used the Least Total Learning 
Time (Group LLT) and the 5 Who Used the Greatest 
Total Learning Time (Group GLT) per Control Test, 

Fall 1969 



Test 

Number 


Group 

Mean 


LLT 

SD 




Group GLT 
Mean SD 


Mean 

Difference 


t-test 


11T° 


693 


41 




678 


34 


15 


0.6613 


16P° 


693 


41 




671 


33 


22 


0.9614 


27T 


679 


86 




652 


64 


27 


0.5597 


39T 


664 


71 




651 


63 


13 


0.3101 


48P 


661 


70 




651 


63 


10 


0.2467 


6 It 


662 


70 




638 


5b 


24 


0.6143 


72T 


696 


24 




649 


42 


47 


2.1500* 


8 OP 


696 


24 




627 


60 


69 


2.3626* 


95P 


701 


22 




627 


60 


74 


2.5827* 


°T = Unit test ; 
*P< .2 


P - 


Post 


test 








AS 


summarized .in 


Table 


2 3 , the 


data 


concerning the 


mean 


difference 


in SATQ 


scores for 


the two 


groups show that on 


tests 



/ 



72, 80 and 95 fairly large differences exist between the means 



O 
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of the two groups favor of Group LLT. Since the standard 
deviation of the mean 6f the latter group is small, the mean 
difference for all three tests is statistically significant. 

It is also interesting to note that although the mean scores 
of the two groups on all tests are within one standard deviation 
(plus or minus) of the mean SATQ score for the class as a whole, 
a closer inspection of the mean and standard deviation of Group 
GLT for tests 61, 80 and 95 reveals the probability that half 
of these students are among those with the lowest SATQ scores 
for the class as a whole. These facts have some practical 
significance for the relationship betvreen the content of the 
Multi-Media Economics Course and learning time. Midshipmen 
who use the least total learning time on tests 72, 80 and 95, 
as a group, have SATQ scores 50 or more points higher than those 
using the greatest total learning time. Inasmuch as the materials 
for the objectives tested in tests 72 and 80 contain many quan- 
titative analyses, students with higher quantitative abilities 
probably find these concepts easier to learn and spend less time 
in the process. This information could be helpful to students 
and instructors when planning learning contracts for Concept 
Area III. 

In addition to the studies of the test scores of Groups 
LLT and GLT, an investigation of the degree of relationship 
between rank in test performance and rank in total learning 
time of the students in these groups was also conducted. The 
results are summarized in Table 24 below. 
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TABLE 24 



Correlation of Rank in Test Scores and 
Total Learning Time per Test for the 5 
Who Used the Least Total Learning Time 
and the 5 Who Used the Greatest Total 
Time (Group GLT) , Fall 1969 


Rank in 
Midshipmen 
(Group LLT) 
Learning 


Test Number 


Group LLT 


Group GLT 


11T° 


.15 


-.80 


16P° 


-.84 


-.04 


27T 


.35 


-.10 


39T 


.02 


-.11 


48P 


.07 


-.53 


61T 


-.48 


-.41 


72T 


-.44 


+ .89 > 


80P 


-.31 


+ .87 * 


95P 


.01 


+ .84 



°T = Unit test; P “ Post test 
*P < .05 

The five correlations of .00 or above indicate a high 
degree of relationship between the variables in the case of 
those particular tests, but only two of these correlations 
are statistically significant. The fact that there is one 
more negative than positive correlation among all those re- 
ported shows a tendency for the variables to be inversely 
related. The r of -.84 for Group LLT shows that the Midship 
men who took the least total learning time for test 16 fre- 
quently made the lowest scores on that test; however, in the 
case of such a small group this correlation is not statis- 
tically significant. Although there are other negative 
correlations reported for this group, they are so low as to 



warrant the conclusion that there is little or no relationship 
between test performance and learning time for Midshipmen who 
take, the least total learning time. 

However, can the same conclusions be drawn regarding 
Group GLT? The correlation of -.80 indicates an inverse 
relationship between total learning time and test performance 5 
on test 11. In other words, those Midshipmen who spend the 

greatest total learning time for test 11 frequently score 

! 

high on that test. On the other hand, the positive correlations 
of .89, .87 and .84 between total learning time and scores on 
tests 72, 80 and 95 respectively show a strong tendency for 
Midshipmen using the greatest learning time for these tests 
to score low on them. It should be noted that only the 
correlations .89 and .87 are statistically significant. The re- 
maining correlations are so low that it can only be concluded that 
in these instances there is little or no relationship between 
the variables, and the little relationship which exists is of an 
inverse nature . 

In summary, it can be said that although a few strong 
relationships between learning time and test scores on some 
tests do exist, as has been noted in the above discussions, 
the majority of the correlations are low. Therefore, the 
only general conclusion which can be drawn for both groups is 

4 

that except for a few instances there is little or no relation- 
ship between the two variables. 

s The rank order for total learning time was from least 
to greatest with the student' taking the least time having a 
rank of 1. The rank order for test scores was from high to 
low with the student making the highest score having a rank of 1* 



4.5*3 Midshipmen Who Completed the Course Six 
WeeKs^ Early 

It. has been reported earlier in this paper 
that twelve Midshipmen, about 32 percent of the class, completed 
the Multi-Media Economics Course by December 9, 1969, approxi- 
mately six weeks before the end of the semester, and that all 
twelve earned a final grade of A or B. This finding raises 
several questions concerning the characteristics of these 
particular students. 

The first question which comes to mind is whether all 
the students who accomplished this feat were always among the 
group of Midshipmen who used the least total learning time per 
test. In the study of the latter group it was found that both 
the Midshipman who used the least total learning time to complete 
the course and the one who used the greatest amount were among 
the 12 who completed the course early. In addition to these 
two, three other Midshipmen who numbered among the group 
taking the least total learning time as well as four from the 
group taking the greatest total learning time, were among the 
twelve. Having students with such a wide range of learning 
time complete the course six weeks before the end of the 
semester leads to the conclusion that the students who work 
most slowly as well as those who work at faster speeds can 
complete both the core and enrichment learnings of the Multi- 
Media Economics Course in one semester or less. How long the 
student takes and his grade seem to be a function of how the 
individual student organizes his schedule and exercises his 
options in order to accomplish his own goals. Table 25 presents 
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the number of "early finishers" per control test in each group — 
those taking the least total learning time or those taking the 
most. All five Midshipmen who required the greatest total 
learning time for test 27 completed the course ahead of 
schedule. 

TABLE 25 

Number of Midshipmen Completing the Multi -Media 
Economics Course Six Weeks Early who Were Among 
the Groups Using the Least Total Learning Time 
(Group LLT) and the Groups Using the Greatest 
Total Learning Time (Group GLT) per Control Test , 



Test Number 


Fall 1969 
Group LLT 


Group GLT 


11T° 


3 


2 


o 

16P 


3 


2 


27T 


2 


C 


39T 


3 


4 


48p 


2 


4 


6 IT 


2 


3 


72T 


2 


2 


80P 


2 


3 


95P 


3 


4 



°T = Unit test; P = Post test 



Another question raised concerning the 12 "early finishers" 
is whether they consistently scored high on all tests. They were 
not, as a group, always among the Midshipmen making the highest 
scores. Eight of the 12 on some one of the tests were among 
those making the highest scores, while four of the 12 were in 
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the group making the lowest scores on one test or another. 
Table 26 presents the ^information for each test. 



TABLE 26 



Number of Midshipmen Completing the Multi-Media 
Economics Course Six Weeks Early Who Were Among 
the Group with High Scores (Group H) and the 
Group with Low Scores (Group L) per Control Test 

Fall 1969 



Test Number 
11T° 
16P° 

27T 

39T 

48P 

61T 



Group H 
2 
2 
2 
0 
1 
3 



Group L 



0 

2 

1 

1 

1 

1 



72T 

80P 

95P 



0 

2 

1 



1 

1 

0 



°T = Unit test; P = Post test 

. To further investigate test performance, the mean scores 
of the 12 who finished early were compared with the means of the 
others in the class who finished at the end of the semester. The 
study shows a difference of one point in four of the tests and 
of only a fraction of a point in the other five tests, with the 
mean scores of the "early finishers" higher than those of the 
rest of the class in all but two instances (see Table 27 ) . The 
one point difference in the means of the groups in the case of 
tests 11, 48 and 95 is statistically but not practically significant. 
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TABLE 27 



Comparison of the Mean Scores of the Midshipmen 
Completing the . Multi-Media Economics Course Six 
Weeks Early (Group I) and of the Rest of the Class 



Taking that Course 
Fall 1969 


(Group II) 


per Control 


Test, 


Test 

Number 


Group I 
(N=12) 


Group II 
(N=27) 


Mean 

Difference 


t-test 


11T° 


36. 6 


35.6 


1.00 


2.3564 


16P° 


28.4 


28.1 


. 30 


0.8315 


27T 


32.0 


31.5 


.50 


0.8527 


39T 


40.3 


40.6 


-.30 


0.4653 


4 8P 


47.0 


46.0 


1.00 


1.3441 


61T 


30.8 


30.1 


.70 


0.6852 


72T 


33.7 


33.8 


-.10 


0.3883 


-8 OP 


45.3 


44.3 


1.00 


1.0791 


95P 


33.6 


32.3 


1.30 


1.9675 


° T = Unit test? P = 
* P <.2 


Post test 







Two other comparisons of the group which completed the 
course ahead of schedule with the rest of the class were made. 
The first concerned the mean total learning time per test and 
explored the question of whether the mean total learning time 
of the group completing the course ahead of schedule is signi- 
ficantly different from that of the group finishing it at the 
end of the semester. For every test the mean total learning 
time for the early finishers is greater than that of the class 
(Table 28) ; however, none of the differences in mean total 
learning time is statistically significant. Although the 
mean difference in total learning time of more than an hour 




/ 
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for tests 48, 80 and 95 might have some practical significance, 
if it pertained to one segment, the fact that the hour or two 

- A 
z 

difference represents only a few minutes additional time for each 
of -some 20 segments (a whole Concept Area) lessens the practical 
significance of this finding. 



TABLE 28 

Comparison of the Mean Total Learning Time 
(Minutes) per Control Test of the Group of 
Midshipmen who Completed the Multi-Media 
Economics Course 6 Weeks Early (Group I) and 
of the Rest of the Class (Group II), Fall 1969 



Test 

Number 


Group I 
(N=12) 


Group II 
(N=27) 


Mean 

Difference 


t-test 


11T° 


391 


366 


25 


0.6236 




(6 1 31" ) 


(6'0C") 


(25") 




16P° 


556 


525 


31 


0.5038 




(9 *16") 


(8*45") 


(31") 




27T . 


336 


282 


52 


1.2667 




(5*34") 


(4*42") 


(52") 




39T 


497 


438 


59 


0.8611 




(8*17") 


(7*18") 


(59") 




48P 


1152 


998 


154 


1.0624 




(19*12") 


(16*38") 


(2*34") 




61T 


328 


291 


37 


0.8144 




(5*28”) 


(4*51") 


(37") 




72T 


443 


421 


22 


0.3653 




(7*23") 


(7*01") 


(22") 




80P 


1088 


966 


122 


0.8189 




(18*08") 


(16*06") 


(2*02") 




95P 


577 


477 


100 


1.1986 




(9*37") 


(7*57**) 


(1*40") 




°T = Unit testj P - Post test 
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The final comparison of the "early finishers" with the 
rest of the class concerns their QPR, SATV and SATQ scores. 

Whether the mean QPR and the mean scores on the SATV 
and SATQ of the two groups are significantly different was 
examined. The findings show that in the case of all three 
variables the difference in the means of the two groups is 
neither practically nor statistically significant (Table 29). 

TABLE 29 

Comparison of the Mean QPR and the Mean SATV 
and SATQ Scores of Midshipmen Who Completed the 
Multi-Media Economics Course Six Weeks Early 
(Group I) and of the Rest of the Class 
(Group II) , Pall 1969 



Index 


Group I (N=l 2 ) 


Group II (N=27 ) 


Mean Diff. 


t-test 


QPR 


2.7 


2.4 


.3 


1.4656 


SATV 


613 


574 


39.0 


1.6485 


SATQ 


668 


6 70 


-2.0 


0.1475 



4.5.4 Summary 

Studies of the performance of the entire group 
of Midshipmen who were enrolled in the Multi-Media Economics Course, 
Pall 1969, as well as of the performance of subgroups, give 
conclusive evidence that the core materials of the course can be 
achieved by Naval Academy students in one semester or less at 
a 90/80 level of performance. Moreover, the results show that 
90 percent of the students in the class complete additional 
enrichment material at an 85 percent level of performance 
within this same time period. Because of the self-pacing 



feature of the course, students who require a greater total 
learning time than their classmates are able to complete the 
course in one semester or less, also. On the basis of these 
findings the course must be evaluated as highly successful 
in promoting efficient learning of economics by Midshipmen. 
These results also imply that the design of the instructional 
materials, which incorporates the sequencing of objectives, 
the conditions^ suitable for the type of learning inherent in 
each objective, and the reinforcement of correct responses, 
is sound. Eliminating any part of the design could result 
in less spectacular performances by Midshipmen. 

4 . 6 Enrichment 

'In the Multi-Media Economics Course the 
Midshipmen along with their instructor determine^ the type 
and amount of enrichment the student would pursue. Many of 
the enrichment objectives were in a different domain (affective) 
from that of the objectives in the core (cognitive) and thus 
were beyond the scope of this contract. Among the enrich- 
ment activities were included seminars, readings, special 
projects and reports which afforded both the student and the 
instructor an opportunity to generate new objectives in the 
affective domain and include them in the contract. There were 
also a few validated instructional packages which showed the 

^These conditions were outlined by Robert M. Gagne 
in The Conditions of Learning , New York: Holt, Rinehart 
and Winston, Inc., 1965. 



instructor whether the student had achieved the associated 
instructional objectives. 

The primary concern was whether or not the 
student elected to take enrichment and whether he completed 
it. The concern was with the affective element rather than 
with a predefined performance related to the content. The 
instructor was given the prerogative of assigning pass-fail 
weights to individual enrichment segments but within the 
bounds of the learning contract negotiated with the students. 
Therefore, an evaluation of the enrichment has not been in- 
cluded in this report. All formal instructional packages 
used in enrichment went through the same validation process 
as core material, and this has been documented in TR-5.39. 
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5. 



media usage 



Throughout the Mul.ti -Media Economics Course the Midshipmen 
can exercise various options concerning media. Three groups of 
core material segments have been prepared in an audio tape- 
workbook format as an alternative to the printed text. Although 
all enrichment is optional, there are choices of films, printed 
segments and computer simulations within the enrichment materi- 
als. Since a number of Midshipmen selected each of the media, 
the impact of each is discussed separately in the sections below. 

5 . 1 Audio Tape -Workbook 

The audio-tape-workbook segments are completely self- 
instructional, and the tapes, which run about 28 minutes, contain 
the same learning material as is in the printed segments. 
Interspersed throughout the taped explanation at appropriate 
points are references to the associated workbook which has the 
same diagrams, practice exercises, and self-test as the printed 
segments. Students choosing the audio tape-workbook format also 
used the student response board for recording their Responses 
to practice exercises and self-tests, so that test data from the 
groups using different media for core materials are easily 
compared. The three series of audio segments have been desig- 
nated Audio A (Segments 8-10, Concept Area I), Audio B (Segments 
22-26, Concept Area II), and Audio C (Segments 43-46, Concept 
Area III) . 

No Midshipman was assigned audio tapes? anyone who 
used this format did so of his own volition. Most of those 



who selected the audio reported they did so because they 
found tapes easy to understand and time-saving. Some also 
reported they preferred listening to reading, while others 
wanted a change in pace. Inasmuch as the same options were 
open to all Naval Academy students in the Multi-Media Economic 
Analysis Course, it may be assumed, then, that just by virtue 
of the fact they made this choice, those who elected to use 
audio are probably different from their classmates in some 
respect. They may prefer at times to learn through the 
sensory channel of hearing rather than of sight, or it may be 
that they have more curiosity. The difference could be 
attributed to any one or to a number of factors. Two Mid- 
shipmen chose all three groups of audio segments; two others 
chose two of the three groups; and all others selected just 
one group. 

To determine whether any difference in QPR, SATV and 
SATQ scores existed between the students who used the audio 
and those who did not, the means of the two groups on each of 
these indices were compared using the t-test (Tables 30, 31, 

32) . The mean SATV score of the students selecting Audio A 
is 46 points lower than that of the rest of the class, a 
difference which is statistically and practically significant. 
Since this mean score is also 35 points lower than the mean for 
the class as a whole (Table 1) , it might be concluded that 
students with SATV scores 30 points or so below the class mean 
will prefer audio. However, considering that this same 
phenomenon does not occur in the dases of Audio E (where the 



one point difference in favor of those selecting audio is 
statistically significant) , nor in Audio C, and inasmuch 
as the scores of the two students who consistently chose audio 
are at the class mean, there is not a strong case for such a 
conclusion. The same is true for the statistically signifi- 
cant difference of .25 points in QPR between the Audio A 
groups. All other statistical evidence shows no significant 
differences in the means of the group for any of the variables 
therefore, it may be concluded that the groups are similar in 
these characteristics . 



TABLE 30 

Comparison of the Mean QPR and the Mean SATV and 
SATQ Scores of the Midshipmen Who Used Audio 
Package A and of the Midshipmen Who Used Comparable 
Printed Segments, Fall 1969 



Index 


Audio A (N=10) 


Printed 


(N=29) 


t~test 




M 


SD 


M 


SD 




QPR 


2.20 


. 30 


2.57 


.55 


2.6561* 


SATV . 


552 


41 


598 


61 


2.6330* 


SATQ 


668 


51 


670 


56 


0.1251 



*P < .2 
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TABLE 31 



Comparison of the Mean QPR and the Mean SATV 
and SATQ Scores of the Midshipmen Who Used 
Audio Package B and of the Midshipmen Who 
Used Comparable Printed Segments, Fall 1969 



Index 


Audio 


B (N=5> 


Printed 


(N=34) 


t- tes t 




M 


SD 


M 


SD 




QPR 


2.26 


.41 


2.50 


.53 


1.2099 


SATV 


573 


52 


572 


110 


4.6417* 


SATQ 


685 


44 


667 


56 


0.7885 



*P < .z 



TABLE 32 

Comparison of the Mean QPR and the Mean SATV 
and SATQ Scores of the Midshipmen Who Used 
Audio Package C and of the Midshipmen Who Used 
Comparable Printed Segments, Fall 1969 



Index 


Audio C 


(N=7) 


Printed 


<N=32) 


t-test 




M 


SD 


M 


. SD 


' 


QPR 


2.46 


.47 


2.48 


.54 


0.0848 


SATV 


611 


61 


581 


59 


1.1861 


SATQ 


687 


69 


666 


51 


0.7635 
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Whenever a student chooses on activity, it is generally 

' +. 

assumed that he does so because he likes it and that he will 
be motivated to perform at a higher level of competence than 
for an assigned one. However, it is possible for him to make 
the selection out of curiosity or for some other reason. If 
this is the case and the person finds he dislikes or is frus- 
trated with his choice, his performance may deteriorate. In 
order to determine whether the audio tape-workbook format had 
any significant impact on the learning of the Midshipmen who 
opted to use it, the percentage of test items referenced to the 
objectives in the audio segments which were answered correctly 
by these Midshipmen was compared with the percentage of items 
referenced to the two segments preceding and to the two seg- 
ments following the audio answered correctly by them. Prior 
to making this comparison, it was necessary to determine the 
difficulty level of the pertinent items on each test by means 
of an item analysis. The latter shows all items to be within 
the same range of difficulty, consequently, differences in 
performance within a test may not be attributed to the fact 
that some questions are more difficult than others. 

The percentages of test items relating to objectives in either 
Audio A, Audio B or Audio C which the Midshipmen answered 
correctly are not significantly different, either statistically 
or practically, from the percentages of correct items refer- 
enced to objectives in the two segments preceding each of the 
audios (Table 33) . 
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TABLE 33 



Comparison of the Percentage of Correct 
Test Items Referenced to Objectives 
in Audio Segments with the Percentage 
of Correct Test Items Referenced to 
Objectives in Two Segments Preceding 
the Audio for Midshipmen Using 
the Audio Segments, Fall 1969 



Test 




NO. Of 
Students 


Non-Audio 

Items 


Audio 
I terns 




t-test 








Segments 


-A- 


Segments 


% 




11T° (Audio 


A) 


10 


6,7 


92.0 


8-10 


91.0 


0.0834 


16P° (Audio 


A) 


10 


6,7 


87.2 


8-10 


90.1 


0.2143 


27T (Audio 


B) 


5 


21A, 21B 


95.6 


22-26 


91.6 


0.2667 


48P (Audio 


B) 


5 


2 1 A , 2 IB 


80.0 


22-26 


85.6 


0.2531 


48P (Audio 


C) 


7 


41,42 


94.0 


43-46 


88.6 


0.3378 



°T = Unit test; P = Post test 

In some instances it was not possible to obtain data 
for the two segments immediately following the audio because 
they were part of a new concept area and were not included in 
the same tests as the audio; therefore, the comparisons are 
limited to Audio A and Audio B (Table 34). The percentage of 
correct test items referenced to objectives in the two segments 
following Audio B was significantly higher from a practical 
but not from a statistical point of view than the percentage 
of correct items referenced to objectives in the segments in 
Audio B. The fact that Segments 23, 29 introduce a new topic may 
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have had a bearing on student performance, or the difference 
may have stemmed from the possibility that the content of 
Segments 22-26 is not especially well-suited to this particular 
format and does not promote a high degree of learning. We can 
only speculate as to the reason for the difference. 



, TABLE 34 

i 

i 

j Comparison cf the Percentage of Correct Test 

j Items Referenced to Objectives in Audio 

Segments with the Percentage of 
Correct Test Items Referenced to the 
Two Segments Following the Audio For 
Midshipmen Using Audio Segments, Fall 1969 



Test 






NO. Of 
Students 


Audio 

items 


Non- Audio 
Items 


t-test 




l 




Segments 


% 


Segments % 




16P° 


(Audio 


A) 


10 


8-10 


90.1 


12,13 97.3 


0.6422 


48P° 


(Audio 


B) 


5 


22-26 


85.6 


28,29 100.0 


0.9434 



°P = Post test 



From the data on Tables 33/ 34 it can be concluded that 
the Midshipmen who elected to use audio tape-workbook segments 
in general do not show great variation in performance from 
segment to segment, regardless of media used. 

The next question investigated was: how does the percen- 
tage of test items, referenced to objectives in the audio segments 
which Midshipmen using the audio answered correctly compare with 
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the percentage of these same items answered correctly by the rest 
of the class? The difference in the percentages is not statis- 
tically significant (Table 35) . Moreover, the difference is so small 
that it has no practical significance, and it may be concluded 
that students who opt to use audio tape- workbook format learn 
the objectives equally as well as the Midshipmen who use printed 
text. 



TABLE 35 

Comparison of the Percentage of Test Items 
Referenced to Objectives in Audio Segments 
Correctly Answered by Midshipmen Using 
Audio- Tape Workbook Packages (Group I) 
and Those Using the Printed Texts 
(Group II) , 





Test 


Group I 


11T° 


(Audio A) 


91.0 


(N=10) 


16P° 


(Audio A) 


90.1 


(N=10) 


27T 


(Audio B) 


91.6 


(N=5) 


48P 


(Audio B) 


85.6 


(N«5) 


48P 


(Audio C) 


88.6 


(N=7> 



°T « unit test; P - post test 



Fall 1969 

Group II t-test 



90.8 


(N=29) 


0.0192 


89.9 


(N=29) 


0. 1667 


95.6 


(N=34) 


0.3175 


85.7 


(N=34) 


0.0064 


93.9 


(N=32) 


0.3942 



a 28 minute playing 
The playing time of’ 

The total 



Although the tapes generally have 
time, some run about three minutes less, 
the tapes should not be confused with learning time. 
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learning time for an audio segment includes not only the time 
used in listening to the tape, but also the time required for 
completing the practice exercises and the self-test plus the 
time for any replaying of the tape which the individual might 
find necessary. Was the total learning time of the Naval 
Academy students with the audio tape-workbook packages greater 
than that of others in the class? For Segments 8-10 (Audio A) , 
Midshipmen using the audio tape- wo rich ook format required on the 
average 17 minutes less total learning time than those using 
the printed text (Table 36). This difference is statistically 
significant because there is a wide range of total learning time 
within the latter group. However, the students using the tapes 
for Audio B (Segments 22-26) on an average used 20 minutes more 
than the rest of the class, but this difference is not statis- 
tically significant. The mean difference for Segments 43-46 
(Audio C) is only eight minutes, with the audio requiring less 
time. For the student who uses all the audio segments, the 
differences would almost strike a balance; but on the whole, 
he would probably take a little less learning time with these 
than with the regular materials. However, the majority of the 
Naval Academy Midshipmen used only one audio package; therefore, 
the fact that the mean learning time for those using Audio B is 
20 minutes longer than the mean time for the users of the 
regular segments could have some influence on the decision of 
Midshipmen interested in selecting audio. 
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TABLE 36 



Comparison of the Mean Total Learning Time 
(Minutes) for Midshipmen Using Audio Tape- 
Workbook Package (Group I) and for Those 
Using Printed Materials (Group II) for 
Segments Prepared in the Two Media, Pall 1969 



Segments 




Group 


I 


Group 


II 


t-test 






Mean 


SD 


Mean 


SD 




8-10 

(Audio A) 


10 


144 

(2*24") 


22 

(22") 


161 

(2*41") 


41 

(41") 


1.7231* 


22-26 
(Audio B) 


5 


155 

(2* 35") 


61 

(l'Ol") 


135 

(2*15") 


52 

(52" ) 


0.6923 


43-46 
(Audio C) 


7 


115 

(l'SS") 


25 

( 25" ) 


123 

(2 ’03") 


46 

(46") 


0.7060 



*P < .2 

The findings reported above lead to the conclusion that 
the use of audio packages did not have an appreciable impact on 
the learning of the objectives in those segments, for the per- 
formance of the Midshipmen selecting these did not vary sig- 
nificantly from segment to segment. On the other hand, the 
students with the audio learned the objectives equally as well 
as those who studied the printed segments and in somewhat less 
total learning time. 
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5.2 Films 



Films used : in the Multi-Media Economic Analysis Course 
were reproductions of a few of the lessons from The American 
Economy , a TV course in economics. Each one contains information 
which has direct bearing on specific objectives in the Multi- 
Media Economic Analysis Course. The films really serve a dual 
purpose: to introduce a topic or to review it. Questions 
referenced to each objective in the film were prepared for 
distribution at the beginning of the film period to direct the 
attention of the students to the relevant material. After the 
film had been shown, the students were to answer the questions 
as a self-test. However, several logistics problems arose in 
connection with the showing of the films (these will be discussed 
in a later section of this report) , and the routine designed for 
showing films and giving questions was not actually followed. 
Nevertheless, in an attempt to determine what effect, if any, 
the films had on the learning of the specific objectives dis- 
cussed in them, a comparison was made of the mean performance 
of viewers and non-viewers on test items referenced to those 
objectives. The results are given in Table 37. 

Although the t-tests indicate statistically significant 
differences for items in three tests, in two of them, the 
difference is a fraction of a point and has no practical sig- 
nificance. In the case of test 27 the difference of one point 
in favor of those seeing the films has practical significance, 
because getting credit for that one item might spell the 
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difference between passing 80 percent or more of the objectives 
and being able to go ahead in the course, or passing fewer than 
80 percent and having to stop for review. 

TABLE 37 

Comparison of the Mean Scores on Test Items 
Related to Films for Midshipmen Who Viewed 



the Film (Group I) and 
View Them (Group 


for 
II) , 


Those Who 
Fall 1969 


Did Not 




Test and 
Film 


N 

I, II 


Group 

Mean 


I 

SD 


Group 

Mean 


II 

SD 


t-test 


16P° ( 10M) 


34, 5 


2.8 


.43 


2.6 


.55 


0.6438 


2 7T° ( 2 1M ) 


29 , 10 


13.0 


.87 


12.1 


1.40 


1.6270* 


39T° ( 38M) 


30,9 


9.4 


.68 


9.8 


.67 


1.2216 


48P (42M) 


32,7 


5.1 


.66 


5.6 


.53 


1.9136* 


72T ( 65M) 


26,13 


2.0 


.00 


1.8 


.38 


1.4771* 


72T (70M) 


30,9 


* 4.9 


.43 


4.9 


.33 


0.1628 


*P < .2 
°T = unit 


testi P 


~ post test 











5.3 Computer Simulations 

Running computer simulations of economic systems is 
another optional enrichment activity in the Multi-Media Economic 
Analysis Course. There are eleven such simulations, and some of 
them are designed to challenge the most advanced first-year 
student. 



78 - 



Six Midshipmen elected not to work any of the simu- 

+ A 

lations? others in the group worked various numbers of them, 
but no one worked all. No conclusions can be drawn concerning 
the effectiveness of the simulations, since the computer was 
usually down when a student was scheduled to run a simulation. 

With down time rates reportedly running as high as 80 percent 
for some students, students typically had to replan their 
schedules several times in order to run one simulation. 

6. STUDENT AND INSTRUCTOR ACCEPTANCE 

6 . 1 Reaction to the Course, as a Whole 

Student acceptance of the Multi-Media Economic 
Analysis Course has been overwhelmingly favorable each semester 
the course has been given. The main reason for the favorable 
reaction is the self-paced aspect which eliminates the necessity 
of attending regularly scheduled class sessions and gives each 
Midshipman full responsibility for his own learning. Every 
Naval Academy student who took the course indicated on the 
"End of Course Critique" that the learning efficiency (the amount 
of information learned per unit of time spend) in this course is 
greater than that in the traditional lecture course. Although 
the instructor continues to like the lecture method and recommends 
that some topics in the course receive less emphasis and that new 
topics be included, he is highly complimentary of the Multi-Media 
Economic Analysis Course because the students in it learn economic 
theory easily and well. He also likes the freedom v which the 
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release from meeting regular classes gives him to work with 
individuals and small groups. He suggests that there should 
be seminars to help the students see the relationship between 
economic principles and current problems. His enthusiasm for 
the course grows as he becomes more familiar with the new role 
which the instructor in the Multi-Media Economic Analysis Course 
must assume. 

Student reaction to the course has been checked through- 
out the semester by means of a Student Critique sheet which the 
Midshipmen were asked to fill out upon the completion of each 
test. Since response was voluntary, not every student returned 
a critique after each test, but about sixty percent replied each 
time. The results have been summarized by test and are included 
in Appendix 4. In addition to commenting on the course each test 
period, the Midshipmen upon completing the course turned in an 
"End of Course" critique, and those who selected the audio tape- 
workbook format for any segments made an appraisal of them. 
Summaries of all responses given on these forms may be found in 
Appendix 4 . 

It is appropriate at this point to compare responses given 
to questions on the test critiques with those given to questions 
posed on the "End of Course" critique. The total number of 
responses to the test critique sheets is 284, to those at the 
completion of the course, 39. Although the questions on both 
are not exactly the same, they concern the same general areas, so 
that there is a validity check. 
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When asked on the test critique about the level of 
difficulty of the materials in the economics course, the 
Midshipmen checked that it was neither. too easy nor too hard 
in 91 percent of all responses. Three percent of the responses 
given indicated that it was too easy, and four percent that it 
was too hard. At the end of the course about 70 percent of the 
Midshipmen reported they had never referred to one of the 
standard texts for additional explanations (detailed references 
to pages were provided in the materials), and 60 percent had 
never consulted their professor for help with any part of the 
course. Twenty percent had studied standard texts less than 
an hour, and about 26 percent had had half-hour conferences with 
the professor to obtain help with some special difficulties. All 
these facts seem to substantiate the conclusions that the course 
content is neither too easy nor too hard for the Midshipmen and 
that the materials are indeed self-instructional. 

In all instances where the Midshipmen were asked to 
compare the approach of the Multi-Media Economic Analysis Course 
with traditional approaches, the students registered a decided 
preference for the self- instructional method of the experimental 
instructional program. Only one percent of the total replies on 
the test critiques showed a preference for the lecture method. 

Two students (less than one percent of the class) checked that 

* t 

the experimental program was less valuable than the lecture 
method, while fewer than ten percent checked that the two 
approaches had about the same value for them. 
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On the "End of Course" critique the Naval Academy 
students were asked to ;make suggestions for improving the 
Multi-Media Economic Analysis Course. Twenty-two, or about 
60 percent, asked that the course be made more relevant to 
present day economics policies and problems through occasional 
non-mandatory seminars, discussions, lectures or films (which, 
incidentally, are included in the course plan). Analysis of 
the test critiques shows that only three students indicated 
that they wanted class discussions, and these three repeated 
their request on several of the critiques. Only once or twice 
was relevancy mentioned on the test critiques; however, the 
fact that over half the students suggested this at the end 
of the course indicates that upon looking back over the course 
as a whole, they recognized a need for this type of activity. 
The lack of such activities is not the fault of the design of 
the course, but rather of the way in which it has been imple- 
mented. In the course plan, there are opportunities to have 
seminars, and some segments (e.g., 47, 57, 71, 99) have been 
set aside for such activities. Thus, since the course plan 
anticipates and encourages such activities, it would be 
possible within the scope of the course to act upon the 
request of the students. 

Although throughout the course eleven students 
mentioned repeatedly that there should be more periods 
scheduled for test taking, only six suggested this at the 
end of the course. Five Midshipmen suggested revising the 
computer simulations, yet twenty-three indicated they had 



to spend time with the professor because of procedural diffi- 
culties with the simulations. During the first few weeks of 
the course* fourteen Midshipmen said they disliked the 
student response board. However, after Concept Area II, no 
one mentioned the student response board,, and on the End of 
Course critique only one person suggested eliminating it. 

There were other suggestions, such as having more audio and 
having more test questions on enrichment which were mentioned 
by only one or two students. All of these are recorded in 
Appendix 4. 

There was no opportunity on the End of Course critique 
for the Midshipmen to note what they liked best, about the 
economics course; however, on the test critique, as mentioned 
before, the self-pacing aspect was mentioned not only most 
frequently but by 30 percent of the students at one time or 
another. Six of those who said the self-pacing aspect was 
tae best thing about the course also wrote that their ten- 
dency to procrastinate and to put off studying until the last 
minute was one of the things they liked least about the experi- 
mental course! Many commented that they liked the clear, 
concise format, but others sometimes found the explanations 
wordy and a few became bored with the sameness of the format. 
Some- disliked one thing, while others liked it; however, the 
course as a whole has been most favorably received by both 
the students and the instructor. 
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6.2 Reaction to Films 



A section of the End of Course critique was devoted 
to the Midshipmen's reactions to films. Their answers were 
somewhat conflicting and indicate that both the films them- 
selves as well as the question of whether they should be 
retained in the course should be reviewed. For example, 27 
Midshipmen answered that the films were effective as a 
supplement to make the course relevant and interesting# yet, 

23 of them wrote that they found the films boring and a waste 
of time. Just as many students mentioned that the films were 
too long, and poorly constructed as said that they were 
realistic, relevant, clear, good presentations of economic 
concepts. There is a dichotomy of opinion which leads to 
the conclusion that the whole question of films should be 
reviewed. 

6.3 Reaction to Audio Tape-Workbook Segments 

Forty percent of the students used at least one of 
the audio tape - workbook series* Two students elected to 
use all three series, and four chose two. Only two of all 
the Midshipmen who selected the audio packages did not finish 
the segments using tape medium. These two returned to the 
printed version, and reported that they were spending much 
more time learning by audio than by reading because they 
could not concentrate on economics when listening to the 
tapes. A few Midshipmen liked the audio because they could 
do other things, e.g., polish shoes, while listening to the 
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tapes; however, this was not the typical reason for liking 
the audio packages . About 80 percent were favorably impressed 
by this medium because it made learning easier for them and 
provided a stimulating variety in learning. They also indicated 
they would like to see more audio options in the course. 

Adverse criticisms included those mentioned earlier in 
this section and the fact that it was difficult to go back 
over materials when using tapes. 

In spite of the fact that the audio tape -workbook 
packages were not widely used by the Midshipmen, they did fill 
a need for those students who prefer to learn using auditory 
senses, and they also provided a change of pace in the course. 
Probably more audio packages should be included in the course, 
but the question of what kind of content is most suited to 
this treatment should be investigated before making any 
additions or changes . 

Although some of the Midshipmen became weary of 
filling out critique sheets, they were on the whole very 
faithful in performing this task,, and their suggestions and 
comments concerning what they did not like were candid, 
.circumspect and valuable. 
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APPENDIX A 



Excerpts from TR-5.37 



(This appendix consists of 
pages 4, 5, 6, 7, 8, 9, 10 
11, 12, 13, 14, 15, 16, 17 
of TR-5.37.) 




employed in the analysis is described in a brief, non-technical 
form in Appendix D and. in more detail in Appendix B. 

Otfr use of the term “validation," then, appears to 
correspond to the term "evaluation" as described by Stolurow 
and employed by Brennan. 

In our planning activities we tend to think of validation 
as a task that precedes "evaluation." We have viewed valida- 
tion as being primarily concerned with the instructional 
effectiveness of the course. On the other hand, we have looked 
upon evaluation as being the process of assessing the inter- 
active effects of the new course and its administrative- 
operational environment, including the acceptability of the 
course and its component parts to faculty and students involved 
in course activities. This "evaluation," of course, includes 
key aspects of the validation process. As in the validation 
activities, student performance data must be obtained. 
Accordingly, student achievement is measured in terms of data 
gathered with criterion-referenced pre and post tests, and data 
are obtained on the amount of study time taken. However, the 
focus is on the question of how well a highly individualized 
learning program can fit within an existing, conventional 
administrative-operational environment. Regardless of the 
instructional effectiveness of the course, an acceptable "fit" 
must be found if the course is to continue to function at the 
Academy . 
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In summary, our use of the term "validation" seems to 
be comparable to Stolurow ar.d Brennan's use of the term 
"evaluation." As used in this report, "evaluation" refers to 
the process of determining the degree of fit between a fully 
operational, highly individualized course and its administrative- 
operational environment. 

III. Basic Methodological Considerations 

In the fully operational economic analysis course, as it 
is currently designed, students will be able to proceed through 
the course at their own pace, make many of their own decisions 
on media usage and optional enrichment areas, and otherwise 
exert a significant degree of control over when and how they 
study. . In other words, the course is highly individualized 
in its intended operational state. It is very desirable that 
the course be evaluated in that highly individualized state. 

A methodological problem can arise if one wishes to 
perform controlled media research during the "evaluation 
semester." An experimental design necessarily calls for 
extensive control of variables. And only when an experiment 
is carefully controlled are its results of value. The type of 
control generally required, however, is directly at odds with 
the concept of a highly individualized course. The very essence 
of a highly individualized course — the flexibility and options — 
presents the problem. It is possible to be faced with the 
alternative of either conducting a carefully controlled exper- 
iment or evaluating a highly individualized course of instruction. 
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Media studies conducted in the past have been 



particularly guilty of following neither alternative well. 

In his review of instructional design and media selection 
factors, Saettler (1968) points out that the predominate type 

i 

,'of media study conducted has been the media comparison study, 
usually yielding a finding of no significant difference. 

He concurs with the observations by Knowlton (1964 ) that 
these studies were not actually research on media because 
their experimental designs did not provide for separating 
the physical characteristics of the media from the sign 
vehicles of the message they carried. Lumsdaine and May £1965} 
and Lumsdaine (1963) in their reviews of media research have 
stressed the inherent limitations of such studies. 

In the present case , it is of critical importance 
that the evaluation apply to the course as it will be operating 
in the future. Any media studies conducted during the evalua- 
tion semester should disturb as little as possible the intended 
course operations. Only then will the results of the evalu- 
ation be useful. And or course, the results of media studies 
which, do not disturb the normal operations will have greater 
value for a curriculum development model than studies con- 
ducted in a temporary artificial atmosphere. 

One last point should be made regarding evaluation 
methodology. There is usually a "shakedown" period after 
a new system is first placed into full operation. During this 
period, it may become necessary to modify the system slightly 
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to overcome an unexpected problem. The previous tryouts of 
the economics course have tended to validate its subsystems 
and its plan of operation. The shakedown period for the 
operational course will be the evaluation semester, which 

I 

begins in September, 1969. The evaluation methodology must 
be such as to permit necessary changes to be made in the 
administration of the course during this period. 

IV. Evaluation Approach 

Evaluation activities may be described as falling into 
four basic categories: 

1. Student performance 

2. student and Instructor acceptance 

3. Course management 

4. General operational environment 

The ultimate measure of the success of a learning 
program is, of course, the performance of the students in 
achieving the learning, objectives. However, the other three 
categories of activities listed above are important in facilitating 
or hampering the achievement of the desired performance. Each of 
the four categories is discussed below in terms of objectives, 
data required, method of data collection, and data analysis. 

It is assumed that a randomly selected student group of between 
thirty and forty students will take the course and that all of 
the selected students will be in their third year at the Academy 
and have had no previous course in economics in college or high 
school. 
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A. Student Performance Evaluation 

Purpose : .The primary purpose of this area of eval- 

uation is to determine the efficiency of instruction. 
Efficiency is measured in terms of learning-time-taken 
an.d achievement scores on criterion-referenced examinations. 

Data Required : Two classes of data are required. 

1. Achievement scores on criterion referenced 
examinations, for each student on each control 
test 

2. Amount of learning time taken by each student 
on each instructional segment. 

Method of Collection : The criterion-referenced 

control tests will be administered to students by the 
course administrator in the area provided for such 
testing. A student will be permitted to take such a 
test as soon as he believes he cam pass it. If he fails 
to achieve criterion performance in two tries, he will 
be required to see his professor, who will make a diagnosis 
and individual learning prescription. This arrangement 
should deter students from taking criterion tests more 
than once, without having studied, in the hopes they might 
pass some by chance. It should also serve to identify 
early the students who may have a serious learning problem 
which needs the diagnostic skills of the instructor. The 
final achievement score obtained by a student on a control 
test will be used as the measure of his "performance.” 
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Time-taken will be recorded on a segment-by-segment 
basis by each student and he will submit his study time 
record to the Course Administrator at the time he takes 
a control test. The total study time accumulated prior 
to the student's passing the control test will constitute 
his learning time-taken for the level of achievement he 
obtained as indicated by his test score. 

Data Analysis : The data analysis will include the 

following: 

a. Group measures 

1) Group achievement per control test 

. range 
. mean 
. median 

2) Group study-time-taken per control test 

, range 
. mean 
. median 

3) Average study-time-taken of bottom and 
top five achievers 

4) Average achievement of five slowest and 
five fastest students in regard to study- 
time-taken 

b. Individual measures 

1) Achievement on individual control tests 

2) Relative standing on achievement tests 

3) Average time-taken per segment 

4) Relative standing in time taken 
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B. Student-Instructor Acceptance Evaluation 

It is important to take into account the attitudes 
and opinions of the students and the instructors involved 
in the multi-media course. Provided the course is 
effective from an instructional learning standpoint, it 
becomes important that the course is "acceptable" to 
those who are involved in it as instructors or students. 
There are different levels of acceptance, ranging upward 
from barely tolerable. The key level of acceptability 
is the level at which ,a student or instructor would be 
willing to continue with a course such as this in the 
next semester. 

Purpose : The purpose of this evaluation area is 

(a) to ascertain if students and/or instructors are 
willing to continue in a course designed as this one, and 

(b) to determine what aspects of the course might be 
changed to increase the acceptability while meeting the 
learning objectives. 

Data Required ; Students' and instructors' comments 
on specific aspects of the course, as well as the course 
in general, will be solicited. The specific aspects of 
the course on which comments will be solicited will 
include each media-materials combination and other 
planned instructional activities. 
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Method of Collection: Critique sheets will be 



issued with instructor materials and student materials 
and will otherwise by made available to the instructors 
and students. These sheets will be sufficiently 
structured so that information will be solicited 
regarding specific and general aspects of the course. 
However, the reply requested will be "open-ended," 
so that no restrictions are placed on obtaining 
possible negative comments* 

Data Analysis : Critique sheets turned in for 

evaluation purposes will be subjected to an analysis to 
determine (a) what aspects of the course were reported 
as best liked and tb) what aspects were negatively 
criticized. Absence of a negative response on an 
aspect of the course will imply that that aspect is 
acceptable. The specific negative and positive 
comments can serve as the basis for revising the course 
administration to increase the general level of 
acceptability. 

C. Course Management 

The course management concerns instructor-student- 
media-materials logistics, schedules, learning prescrip- 
tions, and other aspects of providing for and monitoring 
instructional activities. Course management planning is 
done by the instructors within the context of previously 
established course management policies and procedures. 
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A course Administrator will be responsible for admin- 
istering the plan (though he will perform no instructional 
or guidance function} . Effective course management is 
crucial to the success of a highly-individualized, self- 
paced program. 

Purpose : The purpose of this evaluation area is 

to assess the adequacy of the course management plan and 
its administration and to ascertain what improvements 
might be made. 

Data Required : Information on problems that arise 

regarding course management is needed. This information 
should include a description of the problem, the date it 
first was detected, tile circumstances under which it 
arose, what was done about it, who was involved, and what 
future action seems advisable. 

Method of Collection : The multi-media course 

instructors, the course administrator, and the liaison 

personnel of the contractor will be given a notebook and 

* 

instructions for recording critical incidents. 

Data Analysis : The various notebooks will be brought 

together for an analysis of the events recorded. A deter- 
mination will be made of what action, if any, should be 
taken for each negative event recorded, to prevent its 
reoccurence. 
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D. General Operational Environment 

A highly individualized instructional program cannot 
be designed without regard to the contraints that may be 
imposed by the general environment in which the program 
will be carried out. Similarly, such a course cannot 
be successfully introduced into a conventional educational 
environment without the environment being changed to 
some extent. The "environment," as used here, refers to 
policies, procedures, staffing, facilities, and similar 
factors that constitute the situation within which the 
program, or course, operates. 

Several examples may be given of the kinds of 
"conflicts" between the old and new often created by 
introducing such a course. A grading policy may have 
been established only after long and careful deliberation, 
but when are mid-term grades submitted if a program is 
self-paced and students are at many different places in 
the course? Similarly, a media-materials resource 
center has a set of new requirements to meet when it 
must serve a highly individualized, multi-media course. 

Purpose ; The purpose of this area of evaluation is 
to assess the effects of the course on the operational 
environment and the effects of the environment on the 
operation of the course. This will provide information 
regarding what changes in the course or its environment 
might be advisable to further the learning goals. 



O 

ERIC 



13 - 



Data Required: Information regarding conflicts 



between the multi-media course and its operational 
i environment is necessary. This information should 

include a description of the conflict, the date it was 
first detected, its implications, what was done about 

\ 

• it, and what future action seems advisable. 

Method of Collection : The multi -media course 

instructors, the course administrator, and the contractor 
liaison personnel will be given a notebook and instructions 
for recording operational conflicts. 

Data Analysis : The various notebooks will be brought 

together for an analysis of the events recorded. Recom- 
mendations will be made about what action, if any, should 
be taken to eliminate course-environment conflicts. 

V. Special Media Analyses 

The special media analyses tend to fall into two categories, 
one concerning alternate media and. one concerning the cumulative 
effects of the media configurations used by individual students. 

The primary value of the data gained by these analyses is to 
(a) produce insights into instructor-student -media-operations 
interrelationships and (b) produce hypotheses which may be tested 
in subsequent research activities at the Naval Academy or elsewhere. 
A. Alternate Media 

There are two classes of alternate media in the 

course that are amenable to a comparative assessment.* 

"* The term "assessment" is used because of the judgmental factors 
that will enter into the comparisons. The complexity of the 
situation within a fully operational course does not permit 
the control of variables that would be desired in a "true" 
experimental design. _14„ 



One class is the self-instructional packages. There 
are both, self-instructional audio ‘packages and self- 
instructional printed packages prepared for certain 
instructional topics. It would be possible to compare 
the success of students who use the alternate medium, 
for the same instructional content and behavioral 
objectives. The questions of concern are (1) whether 
or not some students like to learn in a situation like 
this, using one sensory channel as opposed to another, 
and (2) whether or not some students do learn better, in 
situations like this, using one sensory channel as 
opposed to another. 

The second class of alternate media which should 
permit a reasonably acceptable comparative assessment 
to be made concerns the economics simulation models. 
Certain models may be presented in two ways, one which 
uses the computer and one which does not. In the case 

where the computer is used, the student can manipulate 

1 

the model as many times as he wishes in order to see 
what effects are produced. In the alternate case, 
without the computer, the student would get a learning 
package, a key part of which would be the computer 
printouts showing the effects produced when variables 
are manipulated in certain ways. The question of concern 
would be whether or not it was necessary for students to 
actually manipulate a simulation model at a computer 
terminal in order to benefit from the simulation approach 
used in this course. 
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One must resist the impulse to require certain 
students to use; certain media simply to insure that an 
adequate sample of students use each alternate media. 

Not only would this be incompatible with the objective 
of a fully operational course, but the net result might 
be to produce a general negative effect on course activities 
and achievement. However, it would seem appropriate 
to (a) accept volunteers for each of the alternate media 
and (b) where a student is not doing well, generate a 
prescription for an alternate media. Our records, of 
course, would note whether a given alternate medium was 
prescribed or the student volunteered. Given a student 
group of about forty, it would seem reasonable that at 
least twenty-five percent of them would use each alter- 
nate medium under these circumstances. 

Data Required : Achievement scores on criterion- 

referenced examinations, media used, whether media were 
selected voluntarily or prescribed, time-taken per 
segment, and media "acceptance" results, for each student. 

Method of Collection : All of the data required will 

be obtained as part of the evaluation activities described 
in section IV of this report. 

Data Analysis : Students will be grouped according to 

the alternate media they used. An analysis will then be 
made of the characteristics of the students who (a) perform 
better using one medium than the other and (b) perform 
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relatively better than other students on a single given 
medium. The characteristics that will be employed in 
the analysis include a student's SAT-verbal score, his 
SAT-math score, whether or not the alternate medium was 
used by him voluntarily, and the student's achievement 
in other areas of the course. 

B. Cumulative Effects 

Each student's media-usage in the course could 
differ from that of other students. The effects of a 
given sequence of media usage may be cumulative. For 
example, we may find that certain media-usage patterns 
produce an increasingly rapid rate of learning throughout 
the course. Conversely, some media, after prolonged use, 
may tend to induce boredom. We may find that certain media 
have a "halo" effect, either positive or negative, on 
general achievement in the course. 

Data Required ; Achievement scores on criterion- 
referenced control tests, study time-taken, and media 
usage, for each student. 

Method of. Collection : All of the data required 

will be obtained as part of the evaluation activities 
described in section IV of this report. 

Data Analysis : The media-usage of individual students 

will be analyzed to determine if any media-usage patterns 
can be found to account for differences in performance 
levels, and performance trends. As used here, "performance" 
refers to achievement and time taken by the student. 
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APPENDIX B 



Excerpts from TR-5.39 



(This appendix consists of: 
pages 2, 3, 4, 5, 6, 7, 8, 9 , 10 
& 15 of TR-5.39? pages 20-27 

of same report.) 



2. Population and Limitations to Data 



The Multi-Media Economics Course was validated during 
the spring of 1969 on -two groups, one made up of 31 Midshipmen 
enrolled in two of the Economic Analysis classes at the United 
States Naval Academy and a second consisting of 10 male students 
from various universities in the Washington area who came to 
the Educational Technology Center of Sterling Institute to 
take the course. The individuals comprising the latter group 
were chosen because of their similarity to the Midshipmen in 
various characteristics. (See Appendix A.) 

Although it was not possible to obtain a QPR for the in- 
house group, the individual SAT Verbal and SAT Quantitative 
Scores were available. The means of the two groups in each of 
these tests were compared using a t-test. The data in Table 1 
indicate that the mean SAT Verbal scores of the two groups were 
not significantly different; however, the mean SAT Quantitative 
scores of the group were significantly different, the Midship- 
men having a mean score approximately one standard deviation 
above the in-house group. Although the two groups were from 
the same population from the standpoint of SAT V performance, 
they were not from the same population from the standpoint of 
SAT Q performance. The difference in quantitative scores may 
have some practical significance for student performance in 
Concept Area III of the economics course and will be discussed 
later. 



TABLE 1 



Comparison of the Mean SAT Verbal and the Mean 
SAT Quantitative Scores of 
USNA and In-House Students 



Test 


USNA Mean 


In-House Mean 




t 


SAT V 


589 


632 


35 


1.7125 


SAT Q 


674 


590 


35 


3.710* 



*P < . 01 



The data used in this report were collected using student 
response boards which produced punched cards (informally 
referred to as "Dymedia cards"). This data collection process 
is described in Appendix B. 

At least ten Educational Technology Center (ETC) in-house 
student-subjects completed all of the major concept areas of the 
course, with the exception of the last concept area. Concept 
Area IV represents about 10% or less of the course. Ten 



students from the original sample completed Concept Area III, 
but for an assortment of reasons (none of them reltited to the 

experimental course) only two could be present to complete 
Concept Area IV. 

Several junior members of the staff. of the Educational 
Technology Center, who had no previous knowledge of economics, 
also completed the course, including Concept Area IV. The 
data generated by these junior staff were used as informal 
additional information during the preliminary tryouts. Since 
these data were informal in nature , they are excluded from this 
report . 

While the key information in this report concerns the per- 
formance of the Midshipmen during the tryout of the course at 
the Naval Academy, there are a number of reasons why information 
! on the in-house tryouts may be of interest to the reader. 

Accordingly, data on in-house tryouts are given, and often in the 
same table as comparable data from the tryouts at the Naval 
Academy . 

In reviewing such data, particularly if one wishes to make 
comparisons, it must be understood that the Midshipmen were 
presented with a revised version, in the case of many instructional 
segments, of what was used by the in-house students. Moreover, 
some in-house students received a revised version of what other 
in-house students had tried. For example, five students might 
try an instructional segment, which was then revised before the 
five other students were permitted to try it. The data from these 
tryouts by in-house students were grouped together in this report 
for the convenience of the reader. Thus, a mean score for a tesr 
would be based on the actual scores of ten students , though some 
of the students were subjected to a slightly different version of 
instructional materials. 

While the N of the in-house students showed a rapid decline 
for the last part of the course, the N for the Midshipmen 
remained the same throughout the tryout at the Naval Academy. 

On the other hand, while data collection activities where in-house 
students were involved could be controlled, certain data collection 
activities insofar as Midshipmen were concerned were beyond 
complete control and minor data gaps occurred. 

Midshipmen were required to turn in literally hundreds of 
Dymedia cards. Such cards were used to capture their responses 
to learning- imbedded tests, self-tests, and various kinds of 
unit tests , pretests , and post-tests , At various points , some cards 
were lost and others were improperly filled out so that data could 
not be identified or otherwise used. Some students also found 
it difficult to remember to record their study/learning time. 
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In the case of tests, it was not possible to ascertain 
whether the student had acu tally studied all the material 
before taking a segment, unit, or post test. Also, it is 
highly unlikely that all students followed the prescriptions 
and thoroughly reviewed course materials prior to taking a 
make-up test, since scored on the make-up tests in some 
instances were not different from the original scores. 

j The tests for Concept Area XV were scheduled during final 
exam week, so that students had no time for make-ups. In 
addition, one class was told that Post Test 95 (the post test 
for Concept Area IV) would not count, and would not affect the 
students' grades. Consequently, these data defy interpretation, 
for whether scores reflect an effort comparable to that exer- 
cised in other areas or whether they represent just guesses 
cannot be determined. 

Despite these deficiencies, the data available are meaning- 
ful and demonstrate that the Multi-Media Economics Analysis 
Course was an effective instructional system during the spring 
semester of 1969. £/ This document reports the findings of an 
analysis of the available data. 



3. Validation of Course Materials 
3 . 1 Content Validity 

In determining the validity of the instructional 
materials, one of the first questions to be answered was: 

Is the subject matter of the course compatible with that which 
is usually taught in courses of "Economics" or "Principles of 
Economics f The answer to this question called for extensive 
use of expert opinion in the development and writing of the 
materials, and also in the revision of materials. 

The economic content of the materials used in the 
Multi-Media Economics Course was written by approximately 
twenty experts in. the field of economics, including professors 
of economics on the staffs of universities in various parts of 
the United States, as well as economists associated with govern- 
mental or private institutes in both the United States and 
Canada. Following the tryouts, revisions of the materials were 
written by an economist employed full-time by Sterling Institute. 



y The course was revised during the summer of 1969, based 

“ on the detailed empirical data produced in Spring Semester 
tryout. 
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Although the materials were the original work of each economist- 
writer, they were referenced to two well-known texts in economics. 
Economics — ^by McConnell and Economi cs j /by Samuelson. The Multi- 
Media Economics Course differs from the two texts most markedly 
in the arrangement of topics and in presentation, and not in 
economic content. in these two ways, then, by comparison of 
the economic content of the Naval Academy course with the 
content of the two most wide ly-used textbooks in the f ield and 
by review of subject matter experts the content validity of 
the course materials was established. 

3 . 2 Student Performance - Number of Core Objective s 
Learned 



Validation also refers to those activities which 
were designed to determine whether the instructional materials, 
procedures and techniques produced the expected changes in 
behavior in the students at the United States Naval Academy of 
the kind and level anticipated. In other words, validation of 
the course materials, procedures and techniques rested on the 
answer to the question, "Have the students learned?" or even 
more specifically: 

1. Have 90 to 100 percent of the students learned 
75 percent of the core objectives? 

2. Did this learning represent a significant 
increase in knowledge over that which was 
known prior to studying the materials? 

3. Has this learning been accomplished in one 
semester? 

Student performances on various tests given at 
specified intervals during the course have provided the data 
from which the answers to the above questions have been derived. 
Throughout the course the students checked their own learning 
by means of self- tests which were included in the self- 
instructional package at the end of each segment. At predeter- 
mined points in the course, pretests, unit tests, and post tests 
were administered in order to assess the individual's progress 
and to diagnose his difficulties. It may be of value at this 
point to summarize briefly the structure and kinds of tests 
used in the economics course. 



ZJ Campbell R. McConnell , Economics , New York, McGraw-Hill 
Book Company , 1969. 

\j Paul A. Samuelson, E conomics , New York, McGraw-Hill 
Book Company, 1967. 
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3.2.1 Segment Salf-test 



Whenever the student completed studying the 
economic content of a segment and the solving of the problems 
embedded in that content, he took the short, self-administered, 
multiple-choice test which accompanied each segment. Each 
question was referenced to one of the objectives which he had 
just learned; every objective discussed in the segment was 
represented. Whenever he chose an incorrect response, he checked 
the prescription given with the test for suggestions for obtain- 
ing additional information related to the particular objective 
he had not learned. Although the segment self-test was not 
graded, it provided each student with immediate feedback concern- 
ing how well he had learned a group of concepts and gave him 
the opportunity for remediation, if necessary. Each student 
turned in his records of performance which were used in the 
revision of materials. (See Appendices A, B, C.) 

3.2.2 Unit Test 



At predetermined intervals in the course, the 
student was graded on his performance on a unit test which 
included questions referenced to some but not all of the terminal 
objectives for a group of segments, as well as questions refer- 
enced to some of the enabling objectives subsumed under each 
terminal objective. (See Figure 1.) Since the course content 
was written in hierarchical arrangement as in Figure 1, it was 
assumed that when a student passed an item referenced to a ter- 
minal such as 2571, he had learned the enabling objectives 
leading to it. For example, if he passed the item related to 
terminal Objective 2571, but failed one of the enabling objec- 
tives, either 2568 or 2538, it was assumed either that the 
student had made an educated guess in answering the question 
related to terminal Objective 2571 and needed to study further 
the materials for Objectives 2568 or 2538, or that the course 
materials were not effectively producing learning and needed 
revision. Students who answered correctly fewer than 90% of 
the items on a unit test had to take an alternate form (Z) of 
the test as a make-up. The Z form of the test contained new 
items referenced to the same objectives as occured on the 
original test. 
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Fig. 1 - Hierarchy Chart for Hypothetical 

Segment S Showing Terminal Objective 
2571 with its Enabling Objectives 
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3.2.3 Pretests 



Prior to attempting any learning in any 
concept area, a pretest to determine what he already knew about 
the area was given to each student. The pretest consisted of 
questions referenced to each core terminal objective in the 
concept area. Students were cautioned to attempt only those 
questions for which they were absolutely certain of the correct 
answer; therefore, items answered correctly would represent 
topics with which the student was thoroughly familiar. The 
extent of his knowledge, however, would be investigated, and 
depending upon those findings he would be given some options, 
such as exploring the area in depth or increasing knowledge 
in one part of the area, etc., to avoid needless repetition. 
This was one method of individualizing instruction. 

3.2.4 Post Tests 



When a student had completed all the core 
learning required in the concept area, he took a post test. 

Like the pretest, the post test included items referenced to 
each core terminal objective in that concept area, and as in 
the case of the unit tests, it was assumed that if the student 
correctly answered a criterion referenced question for a terminal 
objective, he had learned the objectives subsumed under it. 
Whenever a student answered correctly fewer than 90% of the 
items on the post test, he was given a prescription for remedia- 
tion of his weaknesses and was required to take an alternate 
form (2) of the post test. The 2 form was usually the pretest 
for the concept area, but both the order of the questions and 
the order of the distractors had been rearranged. The scores 
on the post tests were used for grading purposes. 

3.2.5 Enrichment Tests 



In addition to the pre-, post and unit tests 
there were two enrichment tests in the course. These were very 
short tests including only criterion referenced items for each 
terminal objective in the enrichment segments, and students 
ensured credit for enrichment segments by passing 90% of the 
related test items. There were no make-up tests for enrichment, 
as it was envisioned that enrichment would frequently involve 
a special project designed cooperatively by the student and his 
instructor. 



Figure 2 illustrates the testing sequence. 
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Fig. 2 - Testing Sequence for Multi -Media Economics Course 



Did the analysis of the scores on the 
various tests indicate that the students had learned economics 
from studying the course materials? The answer to that is an 
unequivocal "yes." Student performance on the post tests, 
since these consisted of criterion referenced questions for 
each terminal objective in the concept area , was the main 
source of data used in answering this question. In view of 
the hierarchical nature of the course, the learning of one 
terminal objective represented not only the learning of the 
terminal objective itself but also of all the enabling objec- 
tives supporting it.' The data in Table 2 show that the 
average number of objectives subsumed under a terminal objec- 
tive did not vary widely from one concept area to another, 
whereas the approximate total number of specified objectives 
in each concept area varied widely. The total number of 
specified core objectives in the whole course was approximately 
875, and in many cases students of necessity learned many more 
implied core objectives. 

TABLE 2 

Average Number of Enabling Objectives Supporting 
Each Terminal Objective in the Core Materials 
and Approximate Total Number of Objectives 
for Each Concept Area 



Concept 






Ave. Number 


Approx. Total 


Area 






Enabling Obj. 


Number Obj. 


I (Post 


Test 


16) 


8 


135 


II (Post 


Test 


48) 


7 


296 


III (Post 


Test 


83) 


7 


344 


IV (Post 


Test 


95) 


6 


93 



Analysis of all post test data for both 
groups considering the course as a whole reveals that approxi- 

°f, entire body of students learned 75% or more 
of the core objectives. However, the breakdown of the data 

by concept area and group (see Table 3 ) gives a more complete 
picture of the performance. 
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TABLE 8 



Correlation of Rank in Test Performance with 
Rank in SAT V, SAV Q and QPR for 
USNA Midshipmen 



Test 


SAT V 


SAT Q 


QPR 


Unit 11 


-.09 


-.07 


. .33 


Post Test 16 


.12 


.16 . 


.07 


Unit 27 


.33 


-.07 


.30 


Unit 39 


.38 


.45 


.44 


Post Test 48 


.01 


.00 


.29 


Unit 61 


.24 


-.09 


.07 


Unit 72 
* 


.25 


-.34 


.25 


Post Test 83 


.08 


.24 


.38 


Post' Test 95 


-.52 


-.05 


.03 



3 . 4 Learning Time 



And now, the question cf how long the students took 
to accomplish this learning should be considered. In this 
report learning time refers to the number of minutes a student 
used in learning one segment of self “instructional material. 
Each segment was designed to require approximately 50 minutes 
for the "C" student in the conventional system to complete; 
however, in light of the data, this amount of time seems 
generous. The difference between the in-house students and 
the Midshipmen in the mean length of time used in learning 
a segment was net very large, nor was the difference in the 
mean learning time for a segment in each concept area very 
great. Table 9 summarizes the mean number of minutes required 
by each group to learn one segment. 



if 



.-as* 



one semester. Had it been planned that a Midshipman could take 
the tests whenever he was ready to do so, undoubtedly some would 
have finished the course in less than one semester. In the 
usual 3 credit college' course , 45 hours are , spent in class, and 
students are expected to spend 90 hours in study outside of class 
for a total of 145 hours course time. The Multi -Media Course 
indluded 73 core segments, each designed to be learned in 50 
minutes. Inasmuch as students used a. mean time of less than 
40 minutes to learn each of 49 segments, the real time required 
to learn the Multi-Media Economics Course, including all simu- 
lations, tests and enrichment, was considerably less than that 
theoretically expected in a 3-hour one semester course. 

In a discussion of the practicality of the course 
materials some mention must be made of the Dymedia Response 
Board which was used by students for recording responses to all 
problems and tests and for obtaining immediate feedback as to 
their correctness. When interviewed about the course, the 
students were enthusiastic about the instant feedback and felt 
they learned because they were forced, when they made an error, 
to reread and to rethink a problem before they could go on. 
However, they found the board cumbersome and noisy, the latter 
being their major complaint. In view of the fact that the 
Dymedia device was effective in promoting learnincr, attention 
is being given to finding a substitute which would promote 
learning but lack the above-mentioned drawbacks. 

3.7 Conclusion 



The course materials can be viewed as "validated" 
because (a) students achieve the pre-established proportion 
of instructional objectives , (b) the students achieve these 

objectives within the time constraints that apply to the 
course, and (c) senior economists have judged the content 
and objectives to be "valid" economics. 



4 . Validation of Tests 

The amount of learning which took place during the valida- 
tion tryouts of the Multi-Media Economics Course was determined 
by analysis of student performance on tests. The analysis of 
the data would be meaningless , if the tests were not valid? 
therefore, an important task has been to establish the validity 
Of the tests, i.e., to show the faithfulness with which the tests 
measure what they purport to measure. Validity also includes 
the reliability and practicality of the tests. 

4.1 Content Validity of the Items 



The content validity of the course materials was 
established by submitting them to the judgment of experts and 
comparing them with outstanding texts in the field. In estab- 
lishing the content validity of test items again the judgment 
of subject matter experts was used. 





When an economist-writer contracted to write course 
materials, he received not only the objectives for each segment 
but also three criterion referenced , multiple- choice test items 
which had been previously v;ritten for each objective in that 
segment. One of the tasks cf the economist-writer was to check 
each item for content validity. Specifically, he answered these 
questions concerning each item: 

(1) Does the item test the objective? 

(2) Does the correct answer paraphrase or repeat the 
objective? 

(3) Are the distractors appropriate for the question 
and are they incorrect? 

(4) is the item stated clearly and correctly? 

If the content of the item needed revision, the economist- 
writer made the necessary changes; otherwise, the items remained 
as originally written. 

Content validity was initially built into test items 
by making each one criterion-referenced. Each item paraphrased 
or was written in language equivalent to that of the objective 
it tested. At the same time# as they critically reviewed the 
objectives for the course, both the Naval Academy and Sterling 
Institute economists reviewed the criterion-referenced test items 
for the objectives and made appropriate changes. Moreover, 

E fior to the validation tryouts at the Naval Academy, the in- 
ouse tryouts of the tests indicated that the items tested the 
objectives. 

4 . 2 Reliability 



Inasmuch as a test cannot be considered valid unless 
it is reliable, several coefficients of reliability were ob- 
tained. The reliability of each test as a whole was established 
using the Kuder- Richardson Formula 20. Table 14 summarizes the 
coefficient of reliability obtained for each test and includes 
both the number of students who took the tests and the number 
of objectives tested. From the data it is apparent that Post 
Test 16 had practically no reliability, and that Unit Test 78 
and Enrichment Tests 16E and 48E had little reliability. One 
factor present in all three which could account in part for the 
low reliability coefficient was the length of these tests. 

All had relatively few items. Some items in these tests were 
poorly written and as a result were ambiguous. During the 
summer 1969 all tests were revised in view of student error and 
the critiques of the Naval Academy instructors. Items were 
rewritten and the length of tests was changed. In some instances 
it was possible to substitute good items which had already been 
tried out by the Academy Midshipmen for poor ones which they had 
also tried. In light of these changes the reliability coefficients 
should increase. 
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TABLE 14 



Coefficients of Reliability for Each Test 
in the Multi-Media Economics Course 



Test 




N Students 


N Objectives 


r 


Pretest 


2 


38 


15 


.67 


11 


18 


36 


37 


.91 


It 


50 


40 


38 


. 87 


M 


85 


33 


30 


. B6 


Unit Test 


11 


39 


18 


.64 


i* 


27 


39 


25 


.79 


M 


39 


37 


38 


.90 


M 


61 


37 


32 


.72 


It 


72 


* 

cn 

in 


36 


.46 


Post Test 


16 


39 


15 


.15 


M 


48 


38 


37 


.64 


11 


83 


50 * 


52 


.67 


(( 


95 


34 


30 


. 81 


Enrichment 


16E 


28 


10 


.46 


tl 


48E 


23 


11 


.48 



* In this N some students who inadvertently repeated 
the test are counted twice . 



Since at least three test items were written for 
each objective, the question of the equivalency of these items 
arose. In other words, did students who answered the A item 
for objective X correctly, also answer the B item and the C 
item correctly? Some of the tests were constructed so that the 
B item for an objective occurred at the beginning of the test 
and the C item came toward the end of the same test. In others, 
the A and B or A and C items were used in the same way. A 
Pearson r was calculated to determine what relationship existed 
between the forms of the test items. Correlation coefficients 
of .72 and .78 for the B and C items, and .76 and .92 for the 
A and B and A and C items were obtained. 
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Because there were many objectives to be learned in 
Concept Area III, Post Test 83 retested terminal objectives 
previously tested in Unit Tests 61 and 72. A little more than 
two weeks elapsed between the original test and retest situa- 
tions so that. the possibility of any recall effecting the 
correlation was not great. A Pearson r of .76 between Unit 
Test 72 and Post Test 83 and of .61 between Unit Test 61 and 
Post Test 83 was obtained. The above-mentioned correlations 
for different forms of the test items indicate some degree 
of equivalency for the items. 

4.3 Item Difficulty and Discrimination index 

! 

• For the usual test, an item difficulty level:!/ of 40 

j to 60 is considered ideal, while an item with a level of 80 is 
f considered very easy, and one with a level below 30 very 

difficult. The Multi -Media Economics Course was designed so 
that all students could successfully learn all core objectives, 
which meant that all students would pass every test item. 
Theoretically, then, the level of difficulty of each item should 
be 1.00. Item analysis of all the tests in the economics course 
indicated that 66% of the items had a difficulty level of 80 
or above. Although the theoretically ideal level was not reached, 
two-thirds of the items approached that level. Table 15 presents 
the percentages of items in each test within the various levels 
of difficulty. 



57 Item difficulty refers to the percentage of students who 
pass the item; item discrimination refers to the number of 
high-scoring students and the number of low-scoring students 
who pass a particular item. 
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TABLE 15 



Percentages of Test Items Found at Each Level 
of Difficulty 

80+, 60-79, 40-59 , below 39 



Test 


% Items/Diff. 


% Items/Diff. 


% Items/Diff. ■ 


% Items/Diff 


Number 


Level 80+ 


Level 60-79 


Level 40-59 


Level ijQ^nw 39 


11 


72 


28 


0 ' 


0 


16 


60 


7 


13 


20 


27 


* 88 


12 


0 


0 


39 


89 


5 


5 


1 


48 


78 


16 


3 


3 


61 


56 


31 


13 


0 


72 


53 


36 


11 


0 ■ 


83 


60 


35 


4 


1 


95 


40 


37 


17 


6 



Theoretically, there would be zero discrimination 
between the number of high scoring students and the number of 
low scoring students who pass an item, in the Multi-Media 
Economics CoXTse, if its design were successful. Item analysis 
of all test data revealed that 54% of the items had a discrimi- 
nation index of 20 or less. Over half of the items were approach- 
ing the ideal. Table 16 presents the percentage of items having 
a discrimination index of 20 or less for each test. 
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TABLE 16 



Percentages of items with Discrimination 
Index of 20 or Less 

Test % Items Disc, index 

Number 20 or less 



11 


39 


16 


47 


27 


44 


39 


66 


48 


59 


61 


53 


72 


19 


83 


62 


95 


63 



4.4 Practicality 

That the tests be practical is an essential requisite 
of validity. The tests and test items of the economics course 
were practical in that they were easily and quickly scored, 
either manually or by machine. The self -tests ( and learning- 
embedded problems were readily corrected with immediate feedback 
from the Dymedia. No test required more than 50 minutes to be 
completed, and this included time for administrative procedures. 
It was possible to prescribe remedial work based on the indivi- 
dual’s performance and designed to help him overcome his unique 
difficulties. With two equivalent forms available for each 
test, a make-up could be given and any gain in learning ascer- 
tained. Although all the tests to date have been objective, the 
instructor has retained the option to give essay type tests. 

That the course has been tried out and students 
have successfully learned the course materials and passed the 
criterion referenced tests add to the practicality of both 
the tests and the course materials. Student reaction to the 
course as revealed in interviews with twenty-five randomly 
selected Midshipmen was enthusiastic and positive. They partic- 
ularly liked being able to learn at their own speeds, and they 
appreciated knowing exactly what they were supposed to learn. 
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Only two students expressed unfavorable criticism. One of 
these, as he talked about the fact that he did not have enough 
time to study the course, realized that his problem was one of 
budgeting his time to include and not to exclude economics . 

The other student said there was too much reading required. 

All students were in agreement that learning in this course 
seemed more efficient and gave them a feeling of independence. 

The instructors, too, were generally favorably 
impressed with the course, though it required them to adjust to 
a new role. They had more time for individual students, but 
they missed the role of lecturer. 

Both students and instructors were most positive in 
declaring that the Multi -Media Economics Course effectively 
taught economics to the Midshipmen, a convincing argument for 
its validity. 



5. Summary 

The validation of the Multi-Media Economics Course, a self- 
paced, self-instructional course, took place at the United states 
Naval Academy and at the Educational Technology Center of Sterling 
Institute during the spring of 1969. Two sections of students 
regularly scheduled for economics at the Academy and a group of 
■10 students from universities in the Washington area participated 
in the tryouts. The data obtained were analyzed to determine 
whether the course materials did promote the expected changes in 
behavior at the predetermined levels and whether the tests and test 
items tested the instructional materials. 

In order to establish the content validity of both the course 
materials and the test items, about twenty subject matter experts 
(economists) wrote the objectives and course materials, and also 
reviewed the' test items for accuracy and suitability of content. 

In addition, the instructional material of the Multi-Media Economics 
Course was compared with the topics treated in two well-known 
economics texts. A high degree of similarity was found to exist 
in the topics discussed and in the points of view; however, the 
examples, presentation and depth of discussion in the Multi-Media 
Course were the original work of the economist-writers. 

The self-instructional materials were effective in promoting 
the learning of economics at the expected level of performance, 
as evidenced by the following: 

Approximately 80% of the students learned 80% or more 
of the core objectives. (The exception to this was the 
performance in Concept Area IV, explained in Table 3.) 
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2. Approximately 90 to 100% of the students learned 75% 
or more of the core objectives. (The exception to 
this was the performance in Concept Area IV. explained 
in Table 3 . ) 

3, The mean difference in post test and pretest perfor- 
mance (learning gain) was statistically, significant , 
at the .01 level of probability. 



4. All students/ regardless of rank in class and previous 
performances/ were able to learn the core objectives. 

5. The' amount of learning time used to complete the course/ 
which was generally recognized as being the equivalent 
of a two-semester course/ was considerably less than 

the 145 hours allotted to a regular one semester 3-credit 
course. Most segments required a mean learning time of 
approximately 40 minutes. 

In addition to establishing the content validity of the test 
items/ it was necessary to check on their reliability. Analysis 
of test results revealed that with the exception of three cf 
them/ all tests were reliable and the three forms of the test 
items were found to have some degree of equivalence. The level 
of difficulty and the discrimination index of the majority of 
the test items were very satisfactory for this course. It should 
be noted that the economics course was designed so that all stu- 
dents could learn all core objectives. Consequently/ the test 
items theoretically should have a difficulty level of 1.00 and a 
discrimination index of 0. All the tests were easily scored and 
administered and could be completed in 50 minutes/ including 
the administration procedures. In addition/ it was pci-lble to 
prescribe remediation on an individual basis / depending on the 
student's test performance. 

Most of the subjects found the course a welcome change in 
pace and were enthusiastic about learning economics by means of 
self-instructional materials. The instructors also found that 
this method brought about effective learning of economics. 

Analysis of the data from the tryouts revealed that the 
Multi-Media Economics Course/ including both instructional 
materials and tests/ effectively produced the learning of 
economics at the expected levels of performance; therefore/ 

this particular set of instructional materials in economics 
may be considered valid. 
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APPENDIX C 



Number 



O 

ERIC 



of Core Segments and Core Objectives 



After the validation tryouts. Spring 1969, the Multi- 
Media Economic Analysis Course was revised. Materials were 
rewritten and in some instances reorganized to eliminate 
unnecessary repetition of objectives and to give a more 
logical sequence to ^hem. Consequently, some segments were 
moved from one concept area to another, objectives were 
rewritten and sometimes a few were dropped. Table X summarizes 
by concept area the number of objectives and segments in the 
Fall 1969 Revision of the Multi-Media Economic Analysis Course. 



TABLE X 

Number of Core Segments and Core Objectives 
in Each Concept Area of the Multi-Media 
Economic Analysis Course, Fall 1969 



Concept Area 


Segments 


Objectives 


I 


11 


122 


II 


27 


192 


III 


25 


20 4 


IV 


12 


77 
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APPENDIX D 



Summaries of Student Critiques 



Student No . 



S‘tt&tfflaA„Y 

STUDENT CRITIQUE SHEET 

Control Test No. HT~ Date ^alf 



1. Did you think the material was generally either too easy or 
too hard? 

I 

GO Yes, generally too easy 
f~5H Yes, generally too hard 

. 0X1 No , on the whole, neither too easy nor too hard 

i 

2. Were there any learning activities for which you felt that 
you did not have the necessary background? 

mt n ° 

pj“j Yes (If yes, please briefly describe them.) 



3. Considering the experience you have had so far with this 
experimental instructional program, hov; would you compare 
it with the conventional lecture method of teaching courses 




I prefer the experimental instructional program. 
I prefer the. conventional lecture method. 




I like the two approaches about the same. 









O 

ERLC 



4, Since your last Control Test, have you had any problem you 



feel 


is s 


Yes 


NO 


m 


r^i 


m 


Ijal 


\T | 


na 


fTfl! 


152 


fo"! 


IS] 


m 


fj*l 


m 


122 


cm 


m 


cm 


□ 



Having enough study time 



Getting help in understanding difficult material 



d&m&Md&d tbs- 



5. 



6 . 



In one sentence, what do you now like best about the 
experimental instructional program? . , 

r cf «*<& ******* - ** * -! 

C&r<CtsC j< rrnft /fifties t easy tesfitdy a ad c/ttrt <ha tes use /ess cJ/t* //tn - & 

St hi •* Z ttifaitsfin^ «•/ 

JEmmattiat: fe**f4*rP “ / 

T9pes *■/ 

hy cmlrttc-'t jpafi ,tr> tf yrette - 1 
77mt Jottf-t ' 

In one sentence, what do you now like least about the 
experimental instructional program? 



Hetftttt^ » f 

Pteftatt// to Qrrtnje - £ 

Prmay oat' amt JtMActttiy casttr -ji , f 
Co, try my i>y me M' a bcoAtt - y 

tutsy to pro etas finite ttttS 
Pymfdtis d* d*t* 6 tot/s a/ert - i 

t>;yf,c*i/t Hxsfiavttfinff ut^tie/pn neS/ey z. 
f/ot caouoA fitytd S6f meats • / 
tkt feu.cd too -/ 

tfaftMif A# — / 



Me c/itS dtsOttr Starts tgAM.Ct.tt. to 
pa>bfe/X£ optodOAy • / 
ftpdASt ? yets err/y ot"C v f*r pats titty 
9 case beefs -/ 

Xhsfettcftens Ate to ‘Can dsnT -/ 

Perez* y*u to Mart aieast sad 
Sfyett's'e Wertz -/ 

Tap mam psey meats ate terfeS m 

One test. -/ 











STUDENT CRITIQUE 


SHEET 


Student No. $3 


Control Test No, 


/<£ Date jv// /f&9 



1. Did you think the material was generally either too easy or 
too hard? 

* 

m Yes, generally too easy 

* — 1 — j 

\~ 0 ~\ Yes, generally too hard 

fj/ ' | No, on the whole, neither too easy nor too hard 

r 

2. Were there any learning activities for which you felt that 
you did not have the necessary background? 

i m No 

\~ 0 ~\ Yes (If yes, please briefly describe them.) 



t 



3. Considering the experience you have had so far with this 
experimental instructional program, how would you compare 
it with the conventional lecture method of teaching courses? 

EE I prefer the experimental instructional program. 

CE I prefer the conventional lecture method. 

nm I like the two approaches about the same. 










4 . 



Since your last Control Test, have you had any problem you 
feel is significant in one or more of the following areas? 
Tloc 4+u.d'e-nii e .kecked /Jo emru/estS -fo Mu's /Lawf/w, 

Yes No 



[£3 MJ 
m i^n 

CS ISD 

m cm 

Q EEl 

m im 

m sa 
GD EEl 

cd m 



Having enough study time 

Meeting with the instructor 

Obtaining instructional materials 

Obtaining equipment (tape recorders, projectors) 

Getting help in understanding difficult material 

Equipment operation 

Reports on your progress 

Arrangements for test-taking 

Other (Please describe) 



bcaA<& bt$ke do ton , S*r»& frwn 



5. In one sentence, what do you now like best about the 
experimental instructional program? 

Wrfk a tat) pace e <t/?/esice ~ £% 

fd/tfaudf ft eafffy iwe&Asfand y* a ft&m 9prc*fU pcm^s fid rtf fi* e/C* t*pks *r 

ptebkmS e*M c&uts**4 fdem /mmedtaJe/y- 3 
Self lii/v fecu?&J<hk, -A- 

e//t /a tfA.fi *1*3 t** 

Mo 9c fittUt frcl daises -* 

Sifts?:*? •' 

6. In one sentence, what do you now like least about the 
experimental instructional program? 

tfetfi/nq tt 

Pfm*dtk - c***y/hy /r?/tc4 wC r f/Y/riyau/ ccct*</s 

Stshedtck. fesfc * p* 3 

dscdi ay pec fin* ,hf di'scats/tyff * K 

P/ff/Ca/fi 4* if tea mactfj. - 3^ 

Bte&S/vt fdjUfr/ttrt - A 

■Zmpttfti/e t* fm'dcuf e/acffy iOhich <puesf*&*} «**» missed - 4 . 

Sc tnc * i tf left* *• / * 

*d*pc<4tde. - / 

y X )*'/Ubt'f;iy Of tn*J*M*P / * 




(Huesfittffs atfitrffn pttfd -/ 
■firm*/' pre /fa-tAiky 



Student No. f/~ 3<S~ 



STUDENT CRITIQUE SHEET 
CorftTrol Test no. 



Date W// /9&? 



1. Did you think the material was generally either too easy or 
too hard? 

| o | Yes, generally too easy 

t 

tn Yes, generally too hard 

I No, on the whole, neither too easy nor too hard 

[ 

j 

2. Were there any learning activities for which you felt that 
you did not have the necessary background? 

| — j — ( Yes (If yes, please briefly describe them.) 

fe&scmf Xhatore, tfAfP, 



3. Considering the experience you have had so far with this 
experimental instructional program, how would you compare 
it with the conventional lecture method of teaching courses? 



rzi 



I prefer the experimental instructional program. 
I prefer the conventional lecture method. 

I like the two approaches about the same. 



ERiC 






4. Since your last Control Test, have you had any problem you 
feel is significant in one or more of the following areas? 

Yes No 



DD SJ 
I ^ i UaJ 
P~1 135 J 
[SI I5SI 
GD I3E) 
m fjyi 
m | 57 i 
ra IMJ 
pe] cm 



Having enough study time 

Meeting with the instructor 

Obtaining instructional materials 

Obtaining equipment (tape recorders, projectors) 

Getting help in understanding difficult material 

Equipment operation 

Reports on your progress 

Arrangements for test-taking 

Other (Please describe) 



In one sentence, what do you now like best about the 
experimental instructional program? 

Cange otr oton syeed —/y 
ftiUtVS / ihitia hse -(& 

Ehm'tntfex unttrcrssxA y ff&ding, effoe and fr+af e t* hln -y 

Sclf-j**c£'d apfireeeft l ntrtxsft tevrytrr+h ¥nsr*rt - £ 

ykt £*tm> toht'h fv limn And. tOhafmt fu front - i- 
Wo ofojjrs -jl 
Time soved-t 
-/ 

In one sentence, what do you now like least about the 
experimental instructional program? 

Ho tom* faints -7 
tktm&k kft times scheduled 
PyntedtA -•hefotoy; run eufof eo^dS -V 
Pi'ffiCnl'b to oneself. -$ ' 

Han. monts end Aod mem - S 
JLitfk **tf tf o. f£ jjuArfirnS ~X 

Too mood JioAt* nurd - X 



KLC 



CmoBftftm ar mt+e it*** &*** -f 



Sefed&t/e* #** &***{& V 

tfofwfffTttt+tfjLt ti t4*dy-{ 

Tfa&hck EjCc&crse* */ 

Sn*e t**>c+pf* *** **t €tf?htn*4'**t****b'- i 
Tt* Hfutihte -l 

Tltynuc4\ rixdCnq *./ 

}t\t{$1~ sfrtdu 2-3 kftaftrfrfasr&nd $biJLu 
Oui ft h£*ft*c e Atuttsf- • | 
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STUDENT CNITIQUE SHEET 

Student No. Af _ » Control Test No. 3*? 7~ Date TzklL 

1. Did you think the material was generally either too easy or 
too hard? 

ra Yes, generally too easy 
f~Xn Yes, generally too hard 

2. ! Were there any learning activities for which you felt that 

you did not have the necessary background? 

1~^t| No 

To t Yes (If yss , please briefly de s c r lb e them * ) 



r 



3. Considering the experience you have had so far with this 
experimental instructional program, how would you compare 
it with the conventional lecture method of teaching courses? 

I prefer the experimental instructional program. 

I prefer the conventional lecture method. 

I like the two approaches about the same. 



EE 
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4, since your last Control Tost, have you had any problem you 
feel is significant in one or more of the following areas? 

Yes No 



m 


UB 


m 


m 


CD 


nn 


CD 


131 


CD 


13X1 


ed 


sn 


nn 


UM\ 


CD 


i 1 ^si 


ca 
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Having enough study time 

Meeting with the instructor 

Obtaining instructional materials 

Obtaining equipment (tape recorders, projectors) 

Getting help in understanding difficult material 

Equipment operation 

Reports On your progress 

Arrangements for test-taking 

Other (Please describe ) 



5. In one sentence, what do you now like best about the 
experimental instructional program? 

Ca N u/trfk afcitva sp&etfj sc/f-pazetl. - A/ 

Waking i'nttrf&ndenHy ~ 5 

(ZvrKsf — / 

Afo tfotsrx 

<£„ /rttfJtff pCt CCZtA 

7e$f% Cms/S tenf~ tut fA mx ■/ 

6. In one sentence, what do you now like least about the 
experimental instructional program? 

f/o Cffmfffattiis -/y J&0 rt;<tny ffT A. 

Teo ftuJ /at fr/?*rs /?$(- *- Exphn* of. i)r*,j>hi- he wtr*ff 

Tkfes fref*tt*cA Arhtc - 3 

e/irt*ow - A. 

Top tndcA t-cp* h'Atfn - 3 - 

fftatfm’f heatf/iif tOftf* ,n ftf** t'etifty drfs *>sf *t* eAf msf 
A> t** I /> w/rr*i r?«e£t*t -/ / 

Pffp'cetif: A- buAjptA /»w. 



I 



Stfmma&'f 

STUDENT CRITIQUE SHEET 

Student No. Af= 5T Control Test No. Date faff 



1. Did you think the material was generally either too easy or 
too hard? 

| &-| Yes, generally too easy 
Hr! Yes, generally too hard 
/ tTT\ No, on the whole, neither too easy nor too hard 

j 

2. Were there any learning activities for which you felt that 
you did not have the necessary background? 

ng n° 

Yes (If yes, please briefly describe them.) 



t 



3. 



Considering the experience you have had so far with this 
experimental instructional program, how would you compare 
it with the conventional lecture method of teaching courses? 



m 

m 



I prefer the experimental instructional program. 
I prefer the conventional lecture method. 

I like the two approaches about the same. 



»■ 
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feel 
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Yes 


No 


ca 


HD 


[EL 


[Si 


CD 
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120 


C23 


imi 


CH 


\Hj 


rxi 


im 


ca 


fiFt 


ca 


LI] 



Having enough study time 

Meeting with the instructor 

Obtaining instructional materials 

Obtaining equipment (tape recorders, projectors) 

Getting help in understanding difficult material 

Equipment operation 

Reports on your progress 

Arrangements for test-taking 

Other (Please describe) 

M Jt&sz ty- 



■i 

-jt 

-/ 



5. In one sentence, what do you now like best about the 
experimental instructional program? 

- p a-c‘> ^ y ^ •&*<€ c ^ -f-lcfcibi i‘ ty Oj firo <yr&- ryi / 

CJiA K. and ptrche ptrs fn fr>>r “ * 

tfe ans<& e & 
tfp cJ«w* 

Gmfmutxtf tzewbte) Of pftri'fcfrtrm j?A£t's*ymer>& A «<■ bPMifUf * # a rututtrtrt»«fn>£ fetopiifytaty - / 

fltuftb seffttenfc for t- fay cup koJ tnif hmt -/ 

6. In one sentence, what do you now like least about the 

experimental instructional program? 
t\fe cw? ytfafafo — /*a 

7fcjt:<r s-fou/t? its pftv^e farf~ yjyutcfs 
Criffaue sArc/i A» {, frsf-t . -i 



Tot? A trie c0rtsumi*f/ 

fot&o itctueSe if /os of Me 

_ epftcf-sf. ttzt/uivf fries t/f- - / 

Pipy' ft fa/zrrpuybe cutts t i Jfe otefsettp'c -A Gf-c.fafTN.ooiw ((/reuse f fa - t 

’Repe.ft’h'on - A- 5/y £ tfrerurf atfhrftf ~ * 

TdfifrrucA tt*uu\p^-yoT'a,*i *9or& qratt? - X 
Too /nucA- ts"f£— fauib .4 

Pi'ffi'c-i'ti- fa tptf- hefp+f 

*tr>OMnfs of. rr>* (-****.( cHtSAjtd. Ay (ttfs - / 
tfoThwiMif * entwe Stmt/** fa M/jf w P stmtttu -t 
O ffcprrfs m progress -/ 
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Ai ft/® iAf/rc^ — / 



STUDENT CRITIQUE SHEET 

Student No. AU S3 Control Test No. l 7 Date y 



1» Did you think the material was generally either too easy or 
too hard? 

I 

I o I Yes, generally too easy 

— 1 

CeO Yes , generally too hard 
. fjjs, | No, on the whole, neither too easy nor too hard 



2. Were there any learning activities for which you felt that 
you did not have the necessary background? 

ns N o 

p/~j Yes (If yes, please briefly describe them.) 

/nJ^ cMjuuu 



3. 



Considering the experience you have had so far with this 
experimental instructional program, how would you compare 
it with the conventional lecture method of teaching courses? 



□n 



1 prefer the experimental instructional program. 



cs 



I prefer the conventional lecture method. 



m 



I like the two approaches about the same. 



O 
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A Since your last Control Test, have you had any problem you 
feel is significant in one or more of the following areas? 

Yes No 



CD 1^-6 1 Having enough study time 

1 7 I Meetirg with the instructor 

CD HZ! Obtaining instructional materials 

CD CEO Obtaining equipment (tape recorders, projectors) 

l~' 7 1 1 Getting help in understanding difficult material 

rZI 1 '$'/ 1 Equipment operation 

I / | \3z, | Reports on your progress 

I \ tUF/'l Arrangements for test-taking 

f *4 - 1 |~ o ~j Other (Please describe) 



- SL 
V 



- & 



5. In one sentence, what do you now like best about the 

experimental instructional program? , . 

-pete fMederr? fe COo*-fca~t‘ aaj/i -ftm&s* i/annf fffl£XAHfr><f 

Ct**c4 fi * , cJpitsL-r €tti y fa* fct/oeo “ 3 
<£*ac£ fo ffae/t 
fEveK-ffh t«o 
Seff-ftrfc 

kbckf-zfs . 

ifnt fictvwn -frtoUtt •ft'* fftfuH* — / > , f t - - 

Cfit/Mte efitems* ecvrrr&frr /c-f us/M Yfrri?ti/&J Can UfK&Atfn-adm&Sfsfi&t* *** 

6. In one sentence, what do you now like leas t about the 

experimental instructional program? v , i 7 , 

fifo ^ O&f jilts 'J 

~¥ Ckm* qcf'Aeff* tOhenn*t*(*y -f 

Twsfaejrfa frpenzs-frsikfr /?£ f/sr f - Ctvfitre -5 _ JLeaffiicf. p>ta(*k»*s in iefmmfs -/ 

**c£c>£ c/nt J/KW tf / rtctfjks Some &*?**'** s a r »f-cfen*. -/ 

ferr** f - </u/f,/7etv - X* 7c sf* -coiM*. fo nmch - / 

CsttwuUv i/rnnla.ht’n -Sf?e**d fcn fH&t&fititef*/ 1 “A 
yfaj£m pv&jf * / 

-0T ce^/ftf ffrtp -/ 
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^pfC/imnA <t*m& ccttpst** feJ /rtfs - / 

" 'f &Anm-frP*n fetSsft’pets art firs ft - / 



Student No 



STUDENT CRITIQUE SHEET 

Control Test No. / 7£. 7 Date f#./f /£&_£_ 



1, Did you think the material was generally either too easy or 
too hard? 

* 

CZ) Yes, generally too easy 
nn Yes, generally too hard 
I No, on the whole, neither too easy nor too hard 

2, , Were there any learning activities for which you felt that 

you did not have the necessary background? 

tus No 

| — / — f Yes (If yes, please briefly describe them.) 




3. Considering the experience you have had so far with this 
experimental instructional program, how would you compare 
it with the conventional lecture method of teaching courses? 



[5 

ca 

□n 



I prefer the experimental instructional program. 
I prefer the conventional lecture method, 

I like the two approaches about the same. 









ERiC 
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. Since your last Control Test, have you had any problem you 
feel is significant in one or more of the following areas? 

Yes No 



ca ns 
m gg 

CD ISEJ 

cei i.m 
czi an 
CD sa 

m i2Ei 

cm ® 
cm eh 



Having enough study time 

Meeting with the instructor 

Obtaining instructional materials 

Obtaining equipment (tape recorders, projectors) 

Getting help in understanding difficult material 

Equipment operation 

Reports on your progress 

Arrangements for test-taking 

Other (Please describe) 



< 



i 

i 

i 



5. In one sentence, what do you now like best 
experimental instructional program? 

Setf-jretceeft j (htiz pfrntfv -// 

No cJass 

f 5 sf-o*ietetfici€*h $****■ uffieAtt e~ -2- 

Tfitit/eKy fetsmtj wmxyout tfuLtt/c-encfutfttf* 

Mo f tnaf e/.amt ~ e(frt* *£ as you at afosut -/ 
t*s fs ' * •/ 

CtrtctfP cit&i/ m* (-£*■ toMt *7 

/tty country's emme-mf -> 

6. In one ’’sentence , what do you now like least about the 
experimental instructional program? 

No contp(oto'b — & 

Lowf * * ’ 

7 jepetftto+i “ ** / j t 

7 toe sho«ft( be * etuotf tfo *■ ***»**™ - 3 

Xba/NfNy fvs4t-htfp#fh** ****** ’* 

Afrdk 0JL- cf/XC&tS£f<rp? 

/til *f**/C* /A frtntat*! § t 

■ Mkrtax! tvrfk: tA by fair 9 ! 

ERIC 



about the 

S* potpfc MiM nsi turtle a a antcutfe 



/ tttrsw fcft'k fftesfrUr 

ffttt efotafks ttnft e/>*4 -/ 
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SmtorftgRy 

STUDENT CRITIQUE SHEET 

Student No. ^ Control Test No. ^ Date ~fct fl flip 

1. Did you think the material was generally either too easy or 
too hard? 

1 * 

| Q | Yes i generally too easy 
nn Yes , generally too hard 
i fa y| No, on the whole, neither too easy nor too hard 

1 

I 

2. Were there any learning activities for which you felt that 
you did not have the necessary background? 

XM NO 

p^rj Yes (If yes, please briefly describe them.) 






3. 



Considering the experience you have had so far with this 
experimental instructional program, how would you compare 
it with the conventional lecture method of teaching courses? 



ES 
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I prefer the experimental instructional program. 
I prefer the conventional lecture method. 

I like the two approaches about the same. 



F# 
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. 4 . 



Since your last control Tost, have you had any problem you 
feel is significant in one or more of the following areas? 



Yes No 



/ p&iftry} t/t*/ Afo't' ^Ut*f/t6Vr , 



m 

GEL 

m 

m 

CD 

m 

m 

m 

cn 



I 1 Having enough study time 
OS Nee ting with the instructor 
I.*?.! Obta ining instructional materials 
feTI Obtaining equipment (tape recorders, projectors) 
Pi f\ Getting help in understanding difficult material 
P iT) Equipment operation 
Va&\ Reports on your progress 
pyTj Arrangements for test-taking 
e>^] Other (Please describe) 



5. In one sentence, what do you now like best about the 
experimental instructional program? 

£sfs-l>3-cmn ■ 1 1'./ -ep ?*ttenc e. - 

JL&gah for* ^ — S 

C !•€£«, C<rHcJfe,<?PZ'{ f» w ft/u ' ** 

Afo‘jPwti$e ~t 

f/etcf 6f ec+n*n*tc‘s »/ 

6. In one sentence, what do you now like leas t about the 

experimental instructional program? * t t 1 . „.y/, J. 

. -* L ithMtmokjfypK* 

StvtoC £€‘/jrr>fn'h ~4mb/ytf0t*s '<5 ^ 

tUrcfurty -A ' , (t/fcCttf h'#€ crt***j> s/xt* 

Gu/f f/ u/MiruS /fits *f 

-vt 

scfLut^/- -f 



/tubes, c/ass <ftsc*tss<™ - / . 

7ffp A&s/ra-c/ - f 

T*0 tft&vtyi 'd*6S ttt /far* £3? -Z, 

A/e It*/ f* yuf/rc A"/ 
To 0 fck t frtnrs -f 
f9ff*r ft t// / rtf JlfcM/*/ 



^ dww**! 1 -/ 

j ; H*/***+<c*t*.t fy j* tr*n*i* * 



, *9*ffc£ut+~f 



STUDENT CRITIQUE SHEET 

Student No, Control Test No, ^EJH Date $> 7 ! }-£>j7b 

1. Did you think the material was generally either too easy or 
too hard? 

% 

EZ] Yes, generally too easy 
CD Yes, generally too hard 

i? / | No, on the whole, neither too easy nor too hard 

j 

2, Were there any learning activities for which you felt that 
you did not have the necessary background? 

xm »° 

j~£~| Yes (If yes, please briefly describe them,) 




3. Considering the experience you have had so far with this 
experimental instructional program, how would you compare 
it with the conventional lecture method of teaching courses? 



□3 

CZ3 

m 



I prefer the experimental instructional program 
I prefer the conventional lecture method, 

I like the two approaches about the same. 






4, Since your last Control Test, have you had any problem you 
feel is significant in one or more of the following areas? 



Yes 


No 
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5. 



6 . 



In one sentence, what do you now like best about the 
rimental instructional program? , , _ 

S it ctrmta, -AI — 

f/<P 5 - » 

TeacfiS.f //rtf <*■*/;-? ett f cks* fy CPyi<rr£ e - 3 

Sc-ff- fesnrtq 

res$,'&;i;4y (Set, s/k/S fM fs tv* ft - ( 

fyhWtMtxW &■ of- tnsfritef&te 



In one sentence, what do you now like least about the 
experimental instructional program? 
tlo ct>m^ (a /Vi fi- S 

$P7>t£ hvrftMtdk -Ccyifut 

JL&tfl CftJm dUboussi&nu — JL.-' 

ffcf fib It 4~0 CoJ'O-C-/- /hs.fa^A-d ft ' 2 * 

fl/ms rftf-bfr>Tftcem/-f 
Pf/rT6/cA./sfe*<^-/ 

Tate* rrrne fans fv s fusty ***** -/ 

L l CsttjAss&o ~f , 
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END OF COURSE CRITIQUE 



Films 



1. Do you think the films were effective as a 
supplement to make the course more relevant 
and interesting? 



Yes 


No 


27 


12 



2. Do you think the films helped you to learn and 
to retain economic theory and/or policy that 
you would not have learned or retained had you 
not attended the films? 

Yes No Some 

15 20 4 



3. Did the material presented in the films dovetail 
well with the material presented in the printed 
or audio self “instructional segments? 

Yes No 1 Sometimes No Answer 

28 6 3 2 

4. How often did films conflict with or contradict 
materials in the regular segments? 

Never Seldom Don * t Know No Answer 

12 15 1 5 



5. Do you think the films helped you do better on the 
unit and post tests? 

Yes No Sometimes 




12 



20 



7 



Films (Continued) 

6. What did you like best about the films? 

(1) Change of pace; variety 5 

(2) Aid in clarifying material; review; 

augmented printed material; reinforce- 
ment of printed segments 12 

(3) Realistic examples; relevant material 7 

(4) Good presentation; logical; clear, 

concise 7 

(5) More interesting than reading 2 

(6) Extra points earned 9 

(7) Nothing 1 

(8) No answer given 4 



7. What did you like least about the films? 

(1) Room where given needed better circulation. 2 

(2) Boring; out of date; waste of time......... 23 



(3) Irrelevant to segment or lessons I was 

studying at that time 7 

(4) Too long; poorly constructed; 

poor narrator 9 

(5) Film test - ambiguous questions; too 

little time for it 2 

(6) Instructed less well than printed segments. 3 

(7) Schedule of films inconvenient 5 

(8) Not enough films 1 

(9) Nothing 1 

(10) No answer given 2 




END OF COURSE CRITIQUE 



Course as a Whole 



8. Do you think that the knowledge you will take away 
from this course will be more or less valuable than 
the knowledge you would take away from an economic 
analysis course taught by the traditional lecture 
method, assuming an average instructor for the latter? 

More Valuable Less Valuable Same No Answer Given 

30 2 3 4 



9. How do you think this course would compare in learning 
efficiency (that is, in the amount of information or 
concepts learned per unit of time spent) with the 
traditional lecture course in economic analysis? 

This cours e is s 

Much More Somewhat More About as Less 

Efficient Efficient Efficient Efficient 



23 16 



10 . How much time throughout the whole course would you 
estimate that you spent referring to or studying 
one of the standard texts? 

None Less than 1 hr. 4-5 hrs . No Answer Given 
27 8 2 2 



11. How much time throughout the course did you spend 
talking to your professor about the course content 
to gain a better understanding of the content? 

None 1/2 hr. or less lh to 3 hrs. No Answer Given 



22 



10 



7 



0 



Course as a whole (continued) 



12. How much time throughout the course did you spend 

talking to your professor about procedural problems 
or difficulties with getting the computer simulations 



to run 


right? 










Very Little 
(less than 




No 


Answer 


None 


H hour) 


2-3 hrs. 




Given 


14 


21 


2 




2 



23. How do you think the material or structure and 
functioning of this course might be improved? 

(1) Occasional no n-manda to ry seminars, 
discussions, or lecture to make 
course relevant to present-day 



economic policies and problems 22 students, 

27 times 

(2) More times scheduled for 

test taking 6 

(3) Eliminate, deemphasize or 

improve computer simulations 5 

(4) Update films 3 

(5) More audios 2 

(6) Review sessions at the *end of 

a concept area 2 

(7) Vary form of printed segments 2 

(8) More questions on enrichment 2 



O 




Audio Critique 

1. For which segments did you use audio materials alone, 
without referring to jprinted segments ? 

8-10 8 only 22-26 43-46 43,44 only 

7 1 3 4 11 

2. For which segments did you use both the audio material 
and the printed packages? 

Hone 9 10 22 23 24 25 26 44 45 46 



14 1 1 11111 111 

3. What things did you like about the audio materials? 

(1) Saves time 7 

(2) A change 4 

(3) Easy to concentrate 3 

(4) Easy to understand 6 

(5) Nothing 1 



4. What things did you dislike about the audio materials? 



(1) Nothing 8 

(2) Boring 2 

(3) Could not concentrate 1 

(4) Takes longer than reading 3 

(5) Explanations not as good? seems to 

dwell on details 3 




( 6 ) 



Could not go back over it easily 



1 



Audio Critique (continued) 



5. Did you find that the audio materials made learning 



easier? If yes, why? 

Yes No 

12 6 

(1) Easier to concentrate 6 

(2) Material followed a more logical sequence. 1 

(3) Easier to listen than to read 1 

(4) Relaxing; novelty 2 

(5) Good supplement to printed segments 1 



6. Did you find that audio materials made learning 
harder? If yes, why? 

Yes No 

2 14 



(1) Did not learn as much 1 

(2) Boring 2 



(3) Could not go back over materials easily.. 1 

7. Do you think that the audio options are valuable 
as a means of providing a stimulating variety of 
learning modes for the student to choose from? 

Yes No 

14 3 

8. Do you think there should be more audio options 
in the course? 

Yes No 



O 




13 



4 



